AnchorDiT: Semantic Anchor-Guided Transformers for Pixel-Space Image Generation

Takeaway

AnchorDiT studies a fundamental issue in pixel-space diffusion models: a single patch granularity makes it difficult to balance global semantics, local details, and computational efficiency at the same time. Large patches provide a macroscopic view and capture global semantics and structural priors over shorter token sequences, but their limited spatial granularity constrains boundaries and textures. Small patches preserve high-frequency information, but they rapidly increase the number of tokens and the cost of global self-attention. The paper summarizes this tension as the granularity dilemma and develops a complete solution around it.

The main innovation of AnchorDiT lies in a tightly integrated system-level design for this problem: cross-attention, SA-RoPE, and register tokens play complementary roles within the same technical pipeline:

The large-patch Semantics Flow extracts multi-level semantic anchors, while the small-patch Fine-grained Flow preserves pixel-level details. Across network depth, geometrically aligned Anchor Connectors allow fine-grained tokens to repeatedly query the corresponding anchors, including the dense semantics carried by VFM-supervised Registers.

This design avoids making redundant fine-grained tokens independently bear the cost of global modeling. AnchorDiT also does not rely on a VAE or RAE in its generation path and therefore bypasses the separate reconstruction bottleneck introduced by image autoencoders. The paper reports main results, component ablations, and representation visualizations on ImageNet at $256\times 256256\backslash times256$ and $512\times 512512\backslash times512$, forming a relatively complete chain of evidence from research motivation and architectural design to empirical validation.

1. Why Does This Problem Matter?

Mainstream high-resolution diffusion models typically compress images into a latent space with a VAE before generation. Latent-space modeling substantially reduces computational cost, but the lossy compression of a VAE introduces an additional information bottleneck and reconstruction error. Some high-frequency textures, small structures, and precise boundaries may be lost in this process. Pixel-space diffusion directly models the natural data distribution and therefore has the potential to bypass this reconstruction-error channel.

Pixel-space Transformers, however, are extremely sensitive to patch size. If the side length of a patch is reduced to $1\mathrm{/}n1/n$ of its original value, the number of tokens grows by $n^{2}n^2$, while the cost of full self-attention grows by approximately $n^{4}n^4$. The model therefore faces a structural tension:

• Large patches produce short sequences and support efficient global modeling, but have insufficient spatial granularity.
• Small patches retain rich details, but many neighboring pixel tokens are highly redundant, making full global attention prohibitively expensive.

The motivation for AnchorDiT is therefore important and well defined: the problem is not simply to select a better patch size, but to reassign the computational responsibilities of "global understanding" and "detail generation."

2. Anchor-Guided Cross-Scale Interactions

AnchorDiT adopts an asymmetric dual-stream architecture:

• The Semantics Flow uses large patches and a shorter token sequence, extracting global structure and multi-level semantics through stacked DiT blocks.
• The Fine-grained Flow uses small patches to preserve the spatial capacity required for boundaries, textures, and small parts.
• Anchor Connectors link the two streams at multiple network depths, passing intermediate representations from the Semantics Flow to the fine-grained branch as semantic anchors.

Figure 2: The architecture of AnchorDiT. The upper Semantics Flow uses large patches and Registers to extract multi-level semantic anchors $s_{i}s\_i$. The lower Fine-grained Flow preserves small-patch features $f_{i}f\_i$ and queries the corresponding anchors through Anchor Connectors at different depths. The right-hand panel shows the SA-RoPE-aligned cross-attention, conditional gating, and FFN inside an Anchor Connector.

In the $ii$-th Anchor Connector, the fine-grained feature $f_{i-1}f\_\{i-1\}$ produces the query, while the semantic anchor $s_{i}s\_i$ from the Semantics Flow produces the key and value:

$$Q=W_Q\mathrm{AdaLN}(f_{i-1}),K=W_K{s}_i,V=W_V{s}_i,Q=W\_Q\backslash operatorname\{AdaLN\}(f\_\{i-1\}),\backslash qquad\; K=W\_Ks\_i,\backslash qquad\; V=W\_Vs\_i,$$

$$f_i=f_{i-1}+{\alpha }_1\mathrm{CA}(Q,K,V)\mathrm{.}f\_i=f\_\{i-1\}+\backslash alpha\_1\backslash operatorname\{CA\}(Q,K,V).$$

Each fine-grained token can therefore attend to semantic anchors according to its content, position, and current generation state. Rather than relying solely on AdaLN-based inter-stream fusion, AnchorDiT uses cross-attention to let fine-grained tokens query semantic anchors explicitly. Because the keys and values come from the compact semantic anchor sequence, the dominant pairwise interaction term changes from $O(N_f^{2}d)O(N\_f^2\; d)$ for full fine-token self-attention to $O(N_f{N}_{a}d)O(N\_f\; N\_a\; d)$ for cross-attention. When the number of semantic anchors $N_{a}N\_a$ is substantially smaller than the number of fine-grained tokens $N_{f}N\_f$, this design provides access to global context at a more controlled cost. This comparison does not represent total model computation, which also includes self-attention in the Semantics Flow, linear projections, and MLPs.

This distinction constitutes the central architectural innovation of AnchorDiT over existing multi-stream methods: it goes beyond AdaLN-based inter-stream fusion by establishing token-wise, content-adaptive queries from fine-grained tokens to multi-level semantic anchors.

3. Three Components Form a Tightly Integrated Design

3.1 Multi-Level Anchor Connectors: Semantic Guidance Throughout Network Depth

AnchorDiT does not perform feature fusion only once at the input or output of the network. Instead, it extracts $s_1,\dots ,s_{n}s\_1,\backslash ldots,s\_n$ from different depths of the Semantics Flow and uses them to guide the Fine-grained Flow sequentially. The fine-grained branch can therefore repeatedly receive semantic guidance as the backbone representation evolves, rather than relying on a single, static semantic injection.

The ablation results support this design. With the Semantics Flow fixed at 24 DiT blocks, increasing the number of Anchor Connectors from 2 to 4 reduces FID from 3.76 to 2.07 and increases IS from 231.0 to 271.4. Increasing the number further to 6 only slightly improves FID to 2.03, while IS falls to 269.6 and the parameter count grows from 676M to 724M. Four connectors therefore provide a favorable performance-parameter trade-off. Although the connector count and parameter count change together, the observed trend remains consistent with the paper's central hypothesis: providing semantic anchoring at multiple network depths benefits fine-grained generation.

3.2 SA-RoPE: Mitigating Geometric Misalignment in Cross-Scale Attention

Standard 2D RoPE typically assigns positions using discrete row and column indices within each token grid. For different patch sizes, the same index, such as $(0,1)(0,1)$, may correspond to entirely different image regions. Directly applying these indices in cross-attention can therefore introduce conflicts into cross-scale matching.

Figure 3: Comparison between standard RoPE and SA-RoPE. In the upper row, the same grid index refers to different physical regions for large and small patches. In the lower row, SA-RoPE uses patch centers in a shared coordinate system, making the relative positions of cross-scale tokens directly comparable.

Scale-Aware Rotary Position Embedding (SA-RoPE) maps the center points of patches at different scales into a shared coordinate system. For a token with patch size $pp$ and original grid location $(i,j)(i,j)$, its new coordinates are

$$i^{\mathrm{\prime }}=\frac{ip+p\mathrm{/}2}{p_{\mathrm{b}\mathrm{a}\mathrm{s}\mathrm{e}}},j^{\mathrm{\prime }}=\frac{jp+p\mathrm{/}2}{p_{\mathrm{b}\mathrm{a}\mathrm{s}\mathrm{e}}}\mathrm{.}i\text{'}=\backslash frac\{ip+p/2\}\{p\_\{\backslash mathrm\{base\}\}\},\backslash qquad\; j\text{'}=\backslash frac\{jp+p/2\}\{p\_\{\backslash mathrm\{base\}\}\}.$$

The relative position between a query and a key is consequently determined by their patch-center locations in a shared coordinate space rather than by two incompatible grid-index systems. SA-RoPE is not a generic technique detached from the main method; it is a targeted design that allows an Anchor Connector to conduct cross-scale attention using comparable coordinates. In the component ablation, adding SA-RoPE reduces FID from 2.24 to 2.19. Although IS decreases slightly from 264.7 to 263.2, the FID improvement provides quantitative support for the geometric-alignment motivation.

3.3 VFM-Supervised Registers: Decoupling Semantic Representation from Spatial Denoising

Large patchified tokens in the Semantics Flow simultaneously contain diffusion noise, spatial position, local image content, and denoising state. If these tokens are forced to align directly with semantic features from a pre-trained Visual Foundation Model (VFM), they must compromise between generative dynamics and representation learning, and may not serve as dedicated semantic carriers.

AnchorDiT therefore introduces a set of learnable Registers without positional embeddings. At the 8th block of the Semantics Flow, these Registers are supervised with DINOv2 features through ${\mathcal{L}}_{\mathrm{R}\mathrm{E}\mathrm{P}\mathrm{A}}\backslash mathcal\; L\_\{\backslash mathrm\{REPA\}\}$. The dense DINOv2 features are uniformly pooled to match the number of Registers before being used as alignment targets. This design creates an explicit division of responsibilities:

• Large patchified tokens continue to handle spatial layout, noise state, and backbone denoising.
• Registers are dedicated to learning dense semantics from the pre-trained VFM.
• The Fine-grained Flow obtains both spatial context and DINOv2-aligned register semantics through Anchor Connectors.

This design also gives register tokens a task-specific reinterpretation. Rather than serving only as redundant activation absorbers, Registers become explicit carriers of dense semantics. In the t-SNE visualization, the large patchified tokens exhibit entangled distributions, whereas the representations of Registers form highly separable clusters. Qualitative comparisons with and without Registers further suggest clearer object semantics and fewer visual hallucinations and structural artifacts. These visualizations serve as auxiliary mechanistic evidence and complement the quantitative ablations below.

Supervising only the Registers clearly outperforms supervising both token types. This result supports the interpretation that the gain arises from the intended decoupling of semantics for representation and generation, rather than merely from adding another feature loss.

The register-count ablation provides further evidence. Using 0, 4, 16, and 64 Registers yields FID scores of 2.19, 2.15, 2.07, and 2.09, respectively. Sixteen Registers achieve the best result among the evaluated register counts. The benefit therefore comes from a moderately sized, specialized semantic channel rather than from unboundedly increasing the number of register tokens.

4. Where Does the Novelty Lie Relative to Existing Paradigms?

The novelty of AnchorDiT is best understood at the level of the overall architectural paradigm rather than any single primitive:

Around the granularity dilemma, the paper provides three interdependent answers:

1. How should computation be divided? Use a short sequence for global semantics and a long sequence for local details.
2. How should the two scales communicate? Use multi-level Anchor Connectors to establish content-adaptive queries.
3. How can this communication remain spatially and semantically reliable? Use SA-RoPE and VFM-supervised Registers to address geometric misalignment and representation entanglement, respectively.

This correspondence among the problem, mechanism, and supporting designs makes the system-level innovation of AnchorDiT clearly identifiable.

5. Experimental Results

The quantitative comparison covers ImageNet class-conditional generation at $256\times 256256\backslash times256$ and $512\times 512512\backslash times512$ with CFG, and all metrics are evaluated using 50K samples:

Four observations are particularly important:

1. The improvement under closely matched reported settings is persuasive. At $256\times 256256\backslash times256$, AnchorDiT-XL and DeCo-XL/16 have comparable parameter counts (676M versus 682M), are both trained for 600 epochs, and use the same sampling budget of $100\times 2100\backslash times2$ NFE. Under this comparable setting, FID improves from 1.69 to 1.61, while IS increases from 304.0 to 314.3.
2. The similarly sized H-model comparison shows a substantial advantage. AnchorDiT-H and PixelREPA-H/16 are closely matched in reported parameter count, training epochs, and NFE. AnchorDiT-H reduces FID from 1.81 to 1.52 while maintaining a comparable IS (317.3 versus 317.2). It also achieves the best $256\times 256256\backslash times256$ FID among the pixel-space methods listed in the paper. In addition, AnchorDiT-XL matches PixelDiT-XL/16 at an FID of 1.61 while obtaining a higher IS (314.3 versus 292.7).
3. Higher-resolution viability is supported. At $512\times 512512\backslash times512$, AnchorDiT-XL reports an FID of 1.92 and an IS of 294.7, compared with the 2.22 FID and 290.0 IS of DeCo-XL/16 in the table. Because the two models are trained for 600 and 340 epochs, respectively, these results are better interpreted as evidence of high-resolution viability than as a controlled architecture-only comparison under identical training budgets.
4. The generation path does not rely on a VAE/RAE decoder. The results are obtained directly in pixel space, thereby bypassing the separate reconstruction bottleneck of an image autoencoder. DINOv2 is used only during training to provide representation supervision for the Registers; it is not part of image decoding.

The component study below is a cumulative ablation. All configurations before the final row apply ${\mathcal{L}}_{\mathrm{R}\mathrm{E}\mathrm{P}\mathrm{A}}\backslash mathcal\; L\_\{\backslash mathrm\{REPA\}\}$ to the large patchified tokens $s_{l}s\_l$; the final step moves the alignment target from $s_{l}s\_l$ to the Registers $s_{r}s\_r$.

From the baseline FID of 2.46 to the full configuration at 2.07, FID progressively improves as components are introduced and the REPA supervision target is adjusted. The paper also provides targeted ablations of the connector count, register count, patch size, and REPA alignment target. Although IS does not increase monotonically at every step, the complete configuration gives the best FID and IS within this cumulative ablation. This experimental organization connects the final performance to the three core mechanisms and strengthens the credibility of the method's interpretation.

6. Compute-Quality Trade-off: Organizing Global Interaction with Compact Anchors

The efficiency goal of AnchorDiT is to avoid full global self-attention over all small-patch tokens while still providing every fine-grained token with access to global semantics. Under the default $256\times 256256\backslash times256$ configuration with large and small patch sizes of 16 and 8, the Fine-grained Flow contains ${32}^2=102432^2=1024$ spatial tokens, whereas the Semantics Flow contains only ${16}^2=25616^2=256$ spatial tokens plus 16 Registers. In terms of token-pair interactions, having 1024 queries attend to this compact semantic anchor sequence is substantially less expensive than constructing complete pairwise attention among 1024 fine-grained tokens.

The results reveal two complementary facts. First, the semantic patch size cannot be excessively coarse, or the anchors themselves will contain insufficient structural information. Second, once the semantic scale is appropriate, finer generation tokens further improve textures and boundaries. The reported 168 GFLOPs of the default 16/8 configuration are approximately one third of the 533 GFLOPs reported for 8/8, showing that the design effectively controls the computational growth of fine-grained modeling. The 16/8 setting provides the best fidelity-cost trade-off among the evaluated configurations with reported FID and IS.

7. How Should the Research Contribution Be Assessed?

From the perspectives of significance, novelty, technical soundness, and empirical validation, the strengths of AnchorDiT can be summarized as follows:

• Clear significance: The paper directly addresses the central tension among global semantics, local details, and computational cost in pixel-space generation.
• Clear core insight: It assigns global modeling and detail generation to tokens at different granularities and then coordinates them through semantic anchoring.
• Systematic methodology: Anchor Connectors, SA-RoPE, and semantic Registers address cross-scale interaction, spatial alignment, and specialization of semantic representations, respectively, while serving the same technical thread.
• A relatively complete chain of evidence: The evaluation covers two image resolutions, multiple model scales, representative pixel-space baselines, step-by-step component ablations, and feature visualizations.
• Competitive results: Without using an image autoencoder in the generation path, AnchorDiT achieves state-of-the-art FID among the pixel-space methods reported in the paper and shows strong high-resolution performance at $512\times 512512\backslash times512$.
• Potential for further extension: The division of responsibilities between semantic anchors and fine-grained queries may motivate extensions to higher resolutions and richer conditioning signals, although these settings are not evaluated in the current paper.

The current experiments focus on ImageNet class-conditional generation with CFG. The conclusions should therefore be understood as a focused validation of pixel-space architectural design, rather than comprehensive coverage of all generation tasks. Within this clearly defined scope, the main results and ablations provide substantial support for the central hypothesis that multi-level semantic anchoring improves fine-grained pixel-space generation. Text conditioning, additional data distributions, variable resolutions, and real-world throughput remain promising directions for future evaluation.

Overall Assessment

AnchorDiT offers a clear and insightful paradigm for pixel-space image generation: the compact semantic anchor sequence performs global modeling, fine-grained tokens preserve local expressiveness, and geometrically aligned, content-adaptive queries connect the two.

Its main value lies at three levels: it clearly identifies the granularity dilemma as an important problem; it proposes a structurally coherent solution; and it provides strong main results and multidimensional ablations that substantially support the effectiveness of that solution. Overall, AnchorDiT is a systematic study of how computation should be organized in pixel-space Transformers, establishing semantic anchoring as an effective paradigm for high-fidelity pixel-space diffusion while offering a compelling route toward global consistency, local fidelity, and controlled computational cost.