category: literaturenote citekey: briedisefficientcorrelationvolumesampling2025 title: Efficient Correlation Volume Sampling for Ultra-High-Resolution Optical Flow Estimation authors: "Briedis, Karlis Martins; Gross, Markus; Schroers, Christopher" year: 2025 date: 2025-00-00 2025 url: "https://arxiv.org/abs/2505.16942v1" zotero_key: U3G2VQEI zotero_storage: A9C83QGT collections: transpordi_planeerimine folder: 001_artiklid firstAuthor: "Briedis, Karlis Martins"

status: converted

Efficient Correlation Volume Sampling for Ultra-High-Resolution Optical Flow Estimation

Karlis Martins Briedis¹,² Markus Gross¹,² Christopher Schroers² ¹ETH Zurich ¨ ²DisneyResearch|Studios

Abstract

Recent optical flow estimation methods often employ local cost sampling from a dense all-pairs correlation volume. This results in quadratic computational and memory complexity in the number of pixels. Although an alternative memory-efficient implementation with on-demand cost computation exists, this is slower in practice and therefore prior methods typically process images at reduced resolutions, missing fine-grained details.

To address this, we propose a more efficient implementation of the all-pairs correlation volume sampling, still matching the exact mathematical operator as defined by RAFT. Our approach outperforms on-demand sampling by up to 90% while maintaining low memory usage, and performs on par with the default implementation with up to 95% lower memory usage. As cost sampling makes up a significant portion of the overall runtime, this can translate to up to 50% savings for the total end-to-end model inference in memory-constrained environments. Our evaluation of existing methods includes an 8K ultra-high-resolution dataset and an additional inference-time modification of the recent SEA-RAFT method. With this, we achieve state-of-the-art results at high resolutions both in accuracy and efficiency.

1. Introduction

Optical flow estimation is a classical low-level computer vision problem that involves estimating dense correspondences between video frames. It has found applications in many downstream video tasks, including action recognition [26], video compression [1], video inpainting [14, 34], and frame interpolation [21]. Many of these tasks are used to process ultra-high-resolution (UHR) content, such as in movie post-production, requiring high-quality flows at their original resolution.

The vast majority of optical flow estimation methods compute cost matching between two images [25, 33], with recent methods adopting dense all-pairs correlation volume sampling based on RAFT [28]. For correlation volume sam-

Figure 1. SEA-RAFT optical flow prediction on ultra-highresolution frames. By estimating optical flow at downsampled resolution, many of the object details are lost. By directly processing high-resolution frames, it fails to estimate large motion. Using our proposed extensions, both large motion and small details can be estimated, additionally achieving 30% faster inference through our efficient correlation volume sampling algorithm.

pling, RAFT either pre-computes the full 4D volume or uses a memory-efficient on-demand cost sampling with a custom CUDA implementation. Both options have been adopted by many subsequent works [10 –12].

However, existing implementations encounter issues with high-resolution inputs. The full volume computation complexity grows quadratically with respect to the number of pixels, making it prohibitive to store with existing hardware. The on-demand sampling achieves memory reduction at the expense of frequent re-computations, and it cannot be implemented efficiently with existing hardware, thus resulting in worse runtime performance.

In this work, we propose a novel correlation volume sampling algorithm that computes a partial cost volume with incremental updates in each recurrent update step. By analyzing the sampling patterns of the full 4D correlation volume in practical applications, we observe that only a small but regular part of the volume is sampled. We leverage these patterns by efficiently computing only the necessary portion of the volume. Our approach combines the strengths of both previous methods, offering fast computations and low sub-quadratic space complexity.

In isolation, the operation performs on par with the default implementation while achieving up to 95% lower memory usage, and it outperforms on-demand sampling by up to 90% while both have similar memory savings compared to the default. When used in RAFT as a drop-in replacement, it reduces the total end-to-end runtime by up to 58% compared to existing methods that are feasible to run with modern hardware. When applied to the recent SEA-RAFT [30], which is already designed for efficiency, it provides approximately 27% runtime reduction.

Additionally, we generate an 8K ultra-high-resolution optical flow dataset based on the BLENDER movie CHARGE and use it to evaluate existing optical flow methods. Our findings show that most methods generally benefit from estimating optical flow at native resolution, but this comes at the cost of capturing large displacements. To address this issue, we propose a test-time extension of the SEA-RAFT method to perform cascaded inference. Without any additional training of the model, it allows reducing the endpoint error for large motion, i.e., magnitude over 128px, by 80%, outperforming all existing methods.

To summarize, we contribute a computationally efficient correlation volume sampling algorithm and perform the first reference-based evaluation of optical flow at ultra-high resolutions. Finally, we propose a cascaded inference-time extension of SEA-RAFT and achieve state-of-the-art results for 8K flow estimation in terms of both accuracy and efficiency.

2. Related Work

Optical Flow Estimation. Traditional optical flow methods have used variants of local and global cost volumes. Here we only cover the methods most closely related to our work and refer to the survey by Zhai et al. [36] for a more complete overview.

FlowNetC [5] and PWC-Net [25] reintroduce the classical concept of cost volume computation for deep learning applications, but process a flattened local cost between source and warped target images. Similarly, several other methods employ cost volume processing but limit processing to the local neighborhood [9, 35].

RAFT [28] revisits the construction of an all-pairs correlation volume and combines it with recurrent iterative refinements that sample matching costs. It proposes two implementations of 4D correlation volume sampling that remain commonly used to date. This approach has subsequently been adopted by many methods, improving estimation of occluded regions [12], encoding matching costs [8, 22], diffusion-based flow updates[17], and other improvements [15, 16, 23, 24, 27, 37]. Recently, SEA-RAFT [30] introduces simple extensions on top of the original RAFT to improve the efficiency and quality.

Other methods [10, 11] distribute the flow updates over multiple levels up to ¹/² resolution. Due to sampling matching costs at high resolutions, the slower on-demand sampling has to be used.

Little work has been done on optical flow estimation at ultra-high-resolution. Xu et al. [31] decompose optical flow estimation into two 1-dimensional operations, enabling processing up to 4K images and provide qualitative results of 4K flow estimation. Similarly, DIP [39] targets efficient inference at high resolution. The Spring benchmark [18] increases the realism and resolution of optical flow evaluation but is limited to HD resolution (1920 × 1080px). GMFlow [32] considers global motion but consequently becomes prohibitively expensive at high resolutions.

Several works [6, 13, 31, 38] focus on reducing computational costs and memory usage by proposing architecture changes that avoid sampling the dense correlation volume but do so at the expense of estimation accuracy.

Unlike prior work, we propose an algorithm to improve the efficiency of all-pairs correlation sampling, which is directly compatible with RAFT and its variants with no compromise in quality. Additionally, we analyze the impact of resolution on the quality of flow prediction of existing methods by evaluating them on high-quality 8K content with available ground truth data.

Operator Efficiency Improvements. Several prior works have focused on improving the computational efficiency of commonly used deep learning operators. FlashAttention [3, 4] proposes a memory-efficient implementation of attention as introduced in the Transformer [29]. Neighborhood Attention Transformer [7] provides an efficient implementation for a local attention computation and proposes a method utilizing the efficient operator.

Similarly, our work proposes a more efficient implementation of the all-pairs correlation volume sampling, matching the exact mathematical operator as defined by RAFT.

3. Correlation Volume Sampling Analysis

In this section, we introduce the correlation sampling problem in more detail, present the default implementation, and provide an analysis of access patterns of the full 4D correlation volume, which form the foundation of our algorithm. The terms cost and correlation volume are used interchangeably.

Figure 2. Overview of the correlation volume sampling. Given a map of correlation between the features of a single source pixel and the features of every pixel in another image, bilinear sampling is used to extract local matching costs around a point of interest. When repeated for every source pixel, the costs are stored in a dense all-pairs correlation volume, where each row and column correspond to a source and target pixel, respectively. This is repeated on multiple levels of scale.

3.1. Problem and Baseline Implementation

Given D-dimensional features F ¹,² ∈ R ^H×W×^D extracted from two images, the visual similarity, or correlation, between two pixels is defined as the dot product of their feature vectors.

A lookup is performed by bilinearly sampling at a local grid around an interest pixel x, with sampling positions defined as integer offsets within radius r. More formally, the sampled correlation at source pixel y is defined as

$$Cr(\mathbf{y}, \mathbf{x}) = { \langle F{\mathbf{y}}^1, F{\mathbf{x} + \mathbf{d}\mathbf{x}}^2 \rangle | \mathbf{d}\mathbf{x} \in \mathbb{Z} \wedge ||\mathbf{d}\mathbf{x}||{\infty} \le r }.$$ (1)

See Fig. 2 for a visualization.

The default implementation of correlation sampling first precomputes a dense 4-dimensional correlation volume C ∈ R H1×W1×H2×W² , where H and W are the height and width of both image features. This is achieved by flattening both images along spatial dimensions, as illustrated on the right side of Fig. 2, and computing the full correlation volume using a single matrix-matrix multiplication

$$\mathbf{C} = \bar{F}^1 \cdot \bar{F}^2,$$ where $\bar{F}^1 \in \mathbb{R}^{[H_1 \times W_1] \times D}, \bar{F}^2 \in \mathbb{R}^{[H_2 \times W_2] \times D}$ . (2)

The output is then reshaped back to 4 dimensions, and bilinear sampling is directly performed on the precomputed C.

In practice, RAFT constructs a 4-level pyramid by average pooling the last two dimensions of C and performs a lookup on the pooled volumes to increase the perceptual window.

Alternatively, a memory-efficient on-demand implementation computes the values of Eq. 1 directly for each source frame pixel. This approach reduces computational and memory complexity and does not require storing any intermediate values. However, in practice, it underperforms compared to the baseline due to operations that are not hardware-friendly and the lack of result reuse between iterations. We refer to RAFT [28] for more details on the sampling procedure.

3.2. Correlation Volume Access Patterns

The lookup grid for each update is defined over a local neighborhood around the current flow estimate. It ensures that the number of sampled cells per row is limited to the local grid size, i.e., (2r + 1)² elements. During iterative updates, the local neighborhood is shifted but remains close to the previous iteration with a significant overlap of the sampled region. Thus, across all flow update iterations, the total number of sampled columns per single source pixel remains low and does not increase with input size.

To empirically verify this, we run the default RAFT implementation on the Sintel [2] optical flow training dataset, while tracking which correlation volume cells are sampled. In Fig. 3[a], we visualize the sampled cells for a single image over all update iterations, represented as a [H¹ × W1] × [H² × W2] volume, where each row shows all matches of a single source pixel. On average, over the whole dataset, only 1.6% of the cells are being sampled. This suggests that we can build a more efficient algorithm that only computes the necessary cells.

However, working with fully sparse matrices and efficiently keeping track of which values are required is challenging. The straightforward way of keeping track of which cells in the cost volume need to be computed is to set respective values in a binary mask. However, that would consume as much memory as the full correlation volume, thus becoming infeasible. Additionally, sparse matrix-matrix multiplications do not typically provide performance improvements unless the sparsity is very high.

Therefore, we suggest representing C in a block sparse format and making decisions about computation per block rather than per pixel. As shown in Fig. 3[b], averaging the number of sampled values per block slightly increases the ratio of cells that need to be computed but remains significantly lower than the full matrix for sufficiently small block sizes. Full numerical results of the sampling patterns are provided in the supplementary material.

Figure 3. Sampling patterns of a single image over all RAFT iterations. Dark regions correspond to cells that have not been sampled while lighter values indicate more sampled values per block.

Reshaped Cost Volume. As the local sampling grid is defined over a 2D neighborhood, when flattened, the values for each row are scattered over multiple column groups and span over many blocks when averaged (Fig. 3[a,b]). This can be improved by pooling the cost volume in a different layout that groups cells based on such 2D patches.

To this end, we reshape the input images into a patchmajor format, where each block is represented continuously in memory, before computing the correlation volume. As shown in Fig. 3[c,d], the block-aware layout significantly increases sparsity without any computational overhead.

4. Efficient Correlation Volume Sampler

Based on our observations on sampling patterns made in Section 3, we propose an efficient algorithm for all-pairs correlation sampling.

An overview of the algorithm is shown in Fig 4. Inputs are pre-processed once per given pair of images and on every iteration of RAFT-based flow updates (typically 4 − 32 iterations) we perform three main steps: a) determining regions of the volume that will get sampled and setting the computation mask; b) computing selected blocks with efficient block sparse matrix-matrix multiplication; c) sampling computed blocks.

We cover the single-level case in detail. To extend to multi-level correlation volumes, we compute every level independently, as in the single-level case, taking averagepooled target frame features F 2

Overall, the algorithm achieves sub-quadratic O(n 1.5 space complexity and is hardware-efficient to enable fast computations. More implementation details, pseudocode, and space complexity derivation are provided in the supplementary material.

4.1. Input Preprocessing

To minimize the number of blocks that need to be computed, we store images in patch-major format as described in Sec. 3.2 with block height B. To simplify the algorithm implementation, we only consider rectangular blocks and use blocks of the same size across all steps.

At first, we pad inputs to a multiple of B, and split the image into B² -sized tiles. Each tile is then independently flattened in row-major order followed by all tiles being flattened in row-major order. This is visualized in Fig. 4 with different-colored arrows. The flattened image is then stored in a contiguous memory block. It speeds up subsequent operations with a small additional space cost.

4.2. Setting Computation Mask

First, we build a mask of which cell blocks in the cost volume need to be computed such that all necessary values can be sampled. To this end, we take the sampling grid positions for the original problem and then divide them with the chosen block size. We then perform the reverse operation of grid sampling - i.e. scattering - to set every cell that needs to be sampled as 1 in the mask while the rest is initialized with 0. By performing the reverse operation of the sampling step, we ensure that all sampled locations have been set and will be computed.

We then compute cumulative sum over the binary computation mask in order find the index of each computed block of the sparse representation in a flattened storage of non-zero entries.

4.3. Sparse Correlation Volume Computation

The mask is used to compute all blocks of the dense correlation volume that correspond to the non-zero entries of the mask, replacing the dense correlation volume computation defined in Eq. 2.

The computation is performed by sampled block sparse matrix-matrix multiplication, for which there exist efficient implementations in multiple libraries. Each block is produced by computing the product of two B⁴×D matrices and thus well optimized in most hardware. In our implementation, we iterate over each row of the computation mask per single CUDA thread block and skip the block computation if the mask has not been set.

Figure 4. Algorithm overview. It consists of input preprocessing and 3 steps per iteration: a) determining blocks that need to be computed; b) computing selected blocks with block sparse matrix-matrix multiplication; c) sampling computed blocks.

4.4. Sparse Volume Sampling

In order to sample the sparse correlation volume, for each looked up cost value we gather the block index, compute the coordinates relative to this block and sample the block locally similar to the baseline.

However, sampling in the sparse volume with a kernel larger than 1, such as bilinear sampling, becomes more difficult as a kernel can span over multiple cell blocks.

To circumvent this problem, we first build a proxy cost volume using single-cell gathering. This proxy volume contains all cells in the convex hull of a single source pixel and enables more complex resampling with different kernels, such as the bilinear kernel used in RAFT-based methods.

4.5. Caching Across Iterations

Since between RAFT iterations there is a significant overlap of blocks that need to be sampled, we optionally cache all precomputed outputs and only compute the update. That is, for iterations after the first one, we remove all entries in the computation mask that already have been computed in one of the previous iterations and only compute an incremental update of correlation volume blocks.

All new blocks are appended to the block cache which is increased if the new set of blocks cannot be added. We use a growth factor of 2 but allocate no more than 5GB more than is needed in the current iteration.

The caching adds a non-negligible memory overhead but allows to achieve significant runtime improvements while remaining feasible to be run on high resolutions.

5. Isolated Sampling Evaluation

To evaluate the proposed algorithm, we first conduct experiments in isolation by only considering the all-pairs correlation sampling.

We run all methods on the final pass of the Sintel [2] benchmark's train set consisting of 1041 samples. The isolated tests are performed by extracting the intermediate query centroids with RAFT at ¹ ⁄⁸ resolution and using randomly-generated feature vectors. For different resolutions, we upsample or downsample the input image. Unless otherwise noted, the experiments are run using PyTorch 2.2.2, CUDA 12.2 and NVIDIA GH200 chip, equipped with 576GB coherent memory and selected to perform measurements at very high resolutions even for memoryintensive methods. Throughout the experiments, we set the block height B = 8.

The correctness was verified with unit tests and observing the endpoint error when used in RAFT. It achieves nearzero 0.03% endpoint error difference compared to the official implementation.

5.1. Isolated Correlation Sampling Results

We measure the runtime and the peak memory consumption, as reported by PyTorch, by running each operation for all dataset image pairs and report the mean and standard deviation over all sample points. As we observe only negligible variance between different runs with the same inputs, we measure a single run per image.

The default setting uses 896 × 2048 input image resolution, with the correlation volume size of (112 × 256)² , 256 feature channels and 32 flow update iterations, matching the official RAFT implementation.

5.1.1. Image Resolution

Primarily, we investigate the impact of the image resolution on the correlation computation and report the results in Fig. 5. As images are scaled uniformly, increasing the

Figure 5. Runtime and peak memory consumption depending on the RAFT input width. Standard deviation is displayed as shaded area, we show memory capacity of different hardware as dotted lines.

RAFT Input Width =	1024px	4096px
Preprocessing	3.6%	2.8%
Setting Computation Mask	19.7%	10.0%
Computing Block Indices	16.5%	5.5%
Updating Cache	36.1%	19.0%
Computing Correlation	10.4%	32.9%
Sampling Volume	13.7%	29.9%
Total Runtime, ms	44.7	124.2
Without Caching, ms	48.2	168.3

Table 1. Runtime breakdown of the steps of our method and block caching ablation.

input width by 2×, increases the number of pixels by 4× and the size of the dense correlation volume by 16×. It can be observed that our method achieves a very good runtime and memory trade-off and is on par to the default implementation in terms of runtime while the on-demand sampling computation significantly underperforms, especially at higher resolutions.

Although the precomputation and caching steps of our approach increase memory usage compared to the ondemand sampling, it remains feasible to run at all resolutions even on consumer hardware. On the other hand, the default implementation shows quadratic memory increase and at 3584 × 8192 resolution requires 719GB just to store the high resolution dense correlation volume, prohibiting to run it with any accessible hardware.

At low resolution 224 × 512 , the absolute computation time and memory usage is very small and both default and our method perform equally well. Exact measurements are provided in the supplementary material.

5.1.2. Other variables

Additionally, we investigate the impact of number of iterations, input feature dimensionality, as well as hardware on the runtime and memory of correlation sampling and provide results in the supplementary document.

Similar to the results described in Section 5.1.1, they show favorable runtime and memory trade-off, and all considered GPU models show results consistent with those shown in Figure 5.

5.2. Component Analysis

We perform an analysis of the different steps performed by our method and provide a runtime breakdown in Tab. 1. It can be observed that a significant part of the runtime is spent on steps that do not involve any compute-intensive operations, such as setting the computation mask and updating the cache, especially at the lower image resolution.

Due to this added overhead that is necessary to perform partial correlation volume computation, it performs better at higher resolutions where correlation computation and sampling takes precedence. Conversely, these overheads do not allow to achieve runtime speedup compared to the computation of the dense volume despite performing fewer floating point operations.

While the result caching introduces a considerable overhead, we show that without it the performance degrades significantly, emphasizing the importance of reusing computations between the steps.

6. Ultra-High-Resolution Evaluation

To evaluate and benchmark optical flow estimation methods on UHR inputs, we render several sequences from the BLENDER movie CHARGE with rendered ground truth displacements. We then propose an inference extension to the SEA-RAFT optical flow estimation method and perform extensive evaluation of the existing methods. We perform these tests with an NVIDIA A100 80GB GPU.

6.1. Dataset

We follow the commonly used benchmarking approach [2, 18] of generating frames and motion vectors from publicly accessible computer-generated movies. We choose BLENDER movie CHARGE as a recent photo-realistic movie

	Highest Input Width	1px error % ↓	EPE px ↓	LM - 1px error % ↓	LM-EPE px ↓	Best Runtime, s	Without Ours, s	Our Improvement
GMFlow [32]	2048 - 1/4	47.0	2.74	86.3	28.67	0.70		n/a
PWC-Net [25]	8192 - 1/1	26.0	6.65	76.1	128.30	0.58		n/a
Flow-1D [31]	4096 - 1/2	23.8	2.23	71.3	31.60	0.78		n/a
DIP [39]	8192 - 1/1	19.8	4.00	51.2	85.62	11.35		n/a
FlowFormer [8]	8192 - 1/1	16.9	3.22	56.4	58.16	16.32		n/a
FlowFormer++ [22]	8192 - 1/1	16.8	3.41	55.3	55.70	16.40		n/a
SCV [13]	4096 - 1/2	19.1	2.83	46.1	46.45	14.52		n/a
HCVFlow [38]	4096 - 1/2	17.9	2.08	54.4	32.92	1.09		n/a
MS-RAFT+ [11]	4096 - 1/2	14.6	1.92	34.5	32.05	4.86	5.48	−11%
RAFT [28]	8192 - 1/1	22.9	45.56	51.1	66.52	4.59	11.00	−58%
CCMR [10]	8192 - 1/1	18.0	4.08	45.7	76.99	18.69	21.43	−13%
CCMR+ [10]	4096 - 1/2	14.5	2.16	37.1	48.99	13.20	14.12	−7%
SEA-RAFT [30]	8192 - 1/1	17.5	17.63	49.2	107.39	1.90	2.61	−27%
SEA-RAFT (Cascaded)	4096 - 1/2 8192 - 1/1	15.8 13.3	1.90 2.70	36.7 31.6	18.61 21.56	0.46 2.00	0.71 2.87	−36% −30%

Table 2. Quantitative evaluation of optical flow estimation methods on a 8K dataset. We list the highest resolution for each that can be run with 80GB of GPU memory. For a full comparison with other methods we also list our ¹/² results. We report the 1px outlier rate, endpoint-error (EPE), both metrics for pixels with large motion (LM, magnitude over 128px) and the best runtime across all variants with and without our improvements. Best result of each metric is highlighted in bold.

that is not used in existing benchmarks. In total, we generate 335 frames at 8192 × 3432px resolution with super-resolved rendered ground at 16K resolution, following [18]. Due to computational reasons, we only consider forward flow. The dataset consists of 332 evaluation pairs and tests for prediction of 9.3B pixels. An example of from the dataset can be seen in Figure 6.

Details on the dataset generation are provided in the supplementary material.

6.2. Cascaded Inference

Our initial experiments indicated that evaluation at high resolutions degrade the performance of estimating large displacements. To mitigate it, we propose a simple cascaded test-time extension of SEA-RAFT that requires no additional training.

Before applying any iterative flow updates, we recursively initialize flows as a lower resolution estimate. More formally, whenever the minimum input dimension is more than 800px, we bilinearly downscale inputs to ¹ 4 resolution, estimate the flow, and initialize the flows with ¹ downscaled outputs (note that flows are updated at ¹ 8 resolution).

It is akin to a multi-resolution version of the warm-start initialization used in RAFT, and unlike MS-RAFT+ [11], it does not require training of multiple resolution modules.

6.3. Evaluation Results

We evaluate several methods on different levels of processing scale by bilinearly downsampling inputs to {1/8; ¹/4; ¹/2; ¹/1} of their resolution and bilinearly upscaling the flow outputs.

Flow accuracy. We report the quantitative evaluation results at the highest feasible scale of each method in Table 2 along with the best runtime across all feasible correlation sampling approaches with and without ours. Our proposed cascaded inference achieves the best large motion scores and the best overall 1 pixel error. For RAFT-based methods our correlation sampling offers significant reduction in the runtime.

We additionally investigate the impact of resolution on the prediction quality and provide a qualitative comparison in Figure 6 and observe that higher resolution inference provides a better reconstruction of fine details. Full quantitative measurements are provided in the supplementary material.

Runtime and Memory. In Table 3, we show the improvement in runtime and peak memory usage of our method in an end-to-end estimation depending on the image resolution.

Across all methods and input resolutions, our method achieves similar runtime as the default sampling with improvement in memory, increasing with the resolution, and similar memory as the on-demand sampling with a significant improvement in runtime.

Additional Datasets. As our performance improvements are agnostic to the input data, we achieve comparable similar results on Sintel [2], KITTI [19], and Spring [18] datasets to their respective resolution scales in Table 3. Full results are provided in the supplementary material.

Figure 6. Qualitative comparison across different evaluation scales with HCVFlow [38], MS-RAFT+ [11], and SEA-RAFT [30]. Image from Charge by Blender Studio.

			Default On-Demand Sampling			Ours
Method	Input Width	Runtime	Memory	Runtime	Memory	Runtime	Memory
RAFT	1024 - 1/8 2048 - 1/4 4096 - 1/2 8192 - 1/1	0.07 0.27	1.85 (+8%) 7.22 (+187%) OOM OOM	0.56 (+405%) 1.06 (+257%) 2.83 (+184%) 11.00 (+140%)	1.67 2.51 5.89 19.40	0.11 0.30 1.00 4.59	1.72 2.51 5.89 19.42
CCMR	1024 - 1/8 2048 - 1/4 4096 - 1/2 8192 - 1/1	0.34	7.94 (+255%) OOM OOM OOM	0.47 (+36%) 1.35 (+18%) 5.22 (+18%) 21.43 (+15%)	1.98 3.59 10.13 36.30	0.34 1.14 4.44 18.69	2.23 4.60 21.12 66.68
MS-RAFT+	1024 - 1/8 2048 - 1/4 4096 - 1/2		OOM OOM OOM	0.44 (+32%) 1.40 (+30%) 5.48 (+13%)	2.85 7.12 24.22	0.33 1.08 4.86	3.84 17.90 53.92
SEA-RAFT	1024 - 1/8 2048 - 1/4 4096 - 1/2 8192 - 1/1	0.03 0.14	1.90 (+7%) 7.30 (+199%) OOM OOM	0.09 (+152%) 0.18 (+68%) 0.62 (+47%) 2.61 (+37%)	1.67 2.00 3.65 10.25	0.04 0.11 0.42 1.90	1.78 2.44 5.24 17.21

Table 3. Runtime (s) and peak memory usage (GiB) of the full optical flow method end-to-end evaluation depending on the correlation computation variant at different scales of the inputs. We report the relative difference compared to our method for columns where they are not performing on par. OOM indicates that the method fails with an out-of-memory error.

7. Conclusion

In this paper, we investigate optical flow estimation for ultra-high-resolution images, with a focus on RAFT-based methods. We first analyze the existing approaches for allpairs correlation volume sampling, their volume sampling patterns in a practical optical flow application and propose an algorithm that utilizes these observations to perform correlation sampling more efficiently. In extensive experiments, we show that our method achieves a good balance between memory consumption and runtime when compared to previous solutions. For high resolutions it can provide more than a 90% improvement in either space or memory compared to the existing solutions and provides a good runtime and memory trade-off.

Additionally, we evaluate existing methods on an 8K resolution dataset and achieve state-of-the-art results on both in accuracy and performance.

In this work, we did not perform extensive low-level fine tuning, kernel fusing, and other optimizations for a fully efficient implementation of the correlation sampling. As shown by our component analysis, large part of the runtime is spent on computationally cheap operations therefore we believe there is large room for further improvement of the performance in future work.

References

[1] Eirikur Agustsson, David Minnen, Nick Johnston, Johannes Balle, Sung Jin Hwang, and George Toderici. Scale-Space Flow for End-to-End Optimized Video Compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8503–8512, 2020. 1
[2] D. J. Butler, J. Wulff, G. B. Stanley, and M. J. Black. A naturalistic open source movie for optical flow evaluation. In European Conf. on Computer Vision (ECCV), pages 611– 625. Springer-Verlag, 2012. 3, 5, 6, 7, 11, 13
[3] Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR), 2024. 2
[4] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Re. FlashAttention: Fast and memory-efficient ´ exact attention with IO-Awareness. In Advances in Neural Information Processing Systems (NeurIPS), 2022. 2
[5] Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick van der ¨ Smagt, Daniel Cremers, and Thomas Brox. FlowNet: Learning optical flow with convolutional networks. In 2015 IEEE International Conference on Computer Vision (ICCV), pages 2758–2766, 2015. 2
[6] Risheek Garrepalli, Jisoo Jeong, Rajeswaran C. Ravindran, Jamie Menjay Lin, and Fatih Porikli. DIFT: Dynamic Iterative Field Transforms for Memory Efficient Optical Flow. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2220–2229, 2023. 2
[7] Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi. Neighborhood Attention Transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6185–6194, 2023. 2
[8] Zhaoyang Huang, Xiaoyu Shi, Chao Zhang, Qiang Wang, Ka Chun Cheung, Hongwei Qin, Jifeng Dai, and Hongsheng Li. FlowFormer: A Transformer Architecture for Optical Flow. In Computer Vision – ECCV 2022, pages 668–685. Springer Nature Switzerland, Cham, 2022. 2, 7, 17, 18
[9] Junhwa Hur and Stefan Roth. Iterative Residual Refinement for Joint Optical Flow and Occlusion Estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5754–5763, 2019. 2
[10] Azin Jahedi, Maximilian Luz, Marc Rivinius, and Andres´ Bruhn. CCMR: High Resolution Optical Flow Estimation via Coarse-To-Fine Context-Guided Motion Reasoning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 6899–6908, 2024. 1, 2, 7, 17, 18
[11] Azin Jahedi, Maximilian Luz, Marc Rivinius, Lukas Mehl, and Andres Bruhn. MS-RAFT+: High Resolution Multi- ´ Scale RAFT. International Journal of Computer Vision, 132 (5):1835–1856, 2024. 2, 7, 8, 17, 18
[12] Shihao Jiang, Dylan Campbell, Yao Lu, Hongdong Li, and Richard Hartley. Learning To Estimate Hidden Motions With Global Motion Aggregation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9772–9781, 2021. 1, 2
[13] Shihao Jiang, Yao Lu, Hongdong Li, and Richard Hartley. Learning Optical Flow From a Few Matches. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16592–16600, 2021. 2, 7, 17, 18
[14] Dahun Kim, Sanghyun Woo, Joon-Young Lee, and In So Kweon. Deep Video Inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5792–5801, 2019. 1
[15] Ao Luo, Fan Yang, Xin Li, and Shuaicheng Liu. Learning Optical Flow With Kernel Patch Attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8906–8915, 2022. 2
[16] Ao Luo, Fan Yang, Xin Li, Lang Nie, Chunyu Lin, Haoqiang Fan, and Shuaicheng Liu. GAFlow: Incorporating Gaussian Attention into Optical Flow. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9642–9651, 2023. 2
[17] Ao Luo, Xin Li, Fan Yang, Jiangyu Liu, Haoqiang Fan, and Shuaicheng Liu. FlowDiffuser: Advancing Optical Flow Estimation with Diffusion Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19167–19176, 2024. 2
[18] Lukas Mehl, Jenny Schmalfuss, Azin Jahedi, Yaroslava Nalivayko, and Andres Bruhn. Spring: A high-resolution high- ´ detail dataset and benchmark for scene flow, optical flow and stereo. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 2, 6, 7, 12, 13
[19] Moritz Menze and Andreas Geiger. Object scene flow for autonomous vehicles. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3061–3070, 2015. 7, 13
[20] Henrique Morimitsu. Pytorch lightning optical flow. https : / / github . com / hmorimitsu / ptlflow, 2021. 13
[21] Simon Niklaus and Feng Liu. Context-Aware Synthesis for Video Frame Interpolation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1701–1710, 2018. 1
[22] Xiaoyu Shi, Zhaoyang Huang, Dasong Li, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, Jifeng Dai, and Hongsheng Li. FlowFormer++: Masked Cost Volume Autoencoding for Pretraining Optical Flow Estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1599–1610, 2023. 2, 7, 17, 18
[23] Austin Stone, Daniel Maurer, Alper Ayvaci, Anelia Angelova, and Rico Jonschkowski. SMURF: Self-Teaching Multi-Frame Unsupervised RAFT With Full-Image Warping. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3887–3896, 2021. 2
[24] Xiuchao Sui, Shaohua Li, Xue Geng, Yan Wu, Xinxing Xu, Yong Liu, Rick Goh, and Hongyuan Zhu. CRAFT: Cross-Attentional Flow Transformer for Robust Optical Flow. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17602–17611, 2022. 2
[25] Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz. PWC-Net: Cnns for optical flow using pyramid, warping,
and cost volume. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8934– 8943, 2018. 1, 2, 7, 17, 18
[26] Shuyang Sun, Zhanghui Kuang, Lu Sheng, Wanli Ouyang, and Wei Zhang. Optical Flow Guided Feature: A Fast and Robust Motion Representation for Video Action Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1390–1399, 2018. 1
[27] Shangkun Sun, Yuanqi Chen, Yu Zhu, Guodong Guo, and Ge Li. SKFlow: Learning Optical Flow with Super Kernels. In Advances in Neural Information Processing Systems, 2022.
[28] Zachary Teed and Jia Deng. RAFT: Recurrent All-Pairs Field Transforms for Optical Flow. In Computer Vision – ECCV 2020, pages 402–419, Cham, 2020. Springer International Publishing. 1, 2, 3, 7, 17, 18
[29] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2017. 2
[30] Yihan Wang, Lahav Lipson, and Jia Deng. SEA-RAFT: Simple, Efficient, Accurate RAFT for Optical Flow. In Computer Vision – ECCV 2024, pages 36–54. Springer Nature Switzerland, Cham, 2025. 2, 7, 8, 13, 17, 18
[31] Haofei Xu, Jiaolong Yang, Jianfei Cai, Juyong Zhang, and Xin Tong. High-Resolution Optical Flow From 1D Attention and Correlation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10498–10507, 2021. 2, 7, 17, 18
[32] Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, and Dacheng Tao. GMFlow: Learning Optical Flow via Global Matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8121– 8130, 2022. 2, 7, 17, 18
[33] Jia Xu, Rene Ranftl, and Vladlen Koltun. Accurate Optical Flow via Direct Cost Volume Processing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1289–1297, 2017. 1
[34] Rui Xu, Xiaoxiao Li, Bolei Zhou, and Chen Change Loy. Deep Flow-Guided Video Inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3723–3732, 2019. 1
[35] Gengshan Yang and Deva Ramanan. Volumetric Correspondence Networks for Optical Flow. In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2019. 2
[36] Mingliang Zhai, Xuezhi Xiang, Ning Lv, and Xiangdong Kong. Optical flow and scene flow estimation: A survey. Pattern Recognition, 114:107861, 2021. 2
[37] Shiyu Zhao, Long Zhao, Zhixing Zhang, Enyu Zhou, and Dimitris Metaxas. Global Matching With Overlapping Attention for Optical Flow Estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17592–17601, 2022. 2
[38] Yang Zhao, Gangwei Xu, and Gang Wu. Hybrid Cost Volume for Memory-Efficient Optical Flow. In Proceedings
of the 32nd ACM International Conference on Multimedia, pages 8740–8749, New York, NY, USA, 2024. Association for Computing Machinery. 2, 7, 8, 17, 18
[39] Zihua Zheng, Ni Nie, Zhi Ling, Pengfei Xiong, Jiangyu Liu, Hao Wang, and Jiankun Li. DIP: Deep Inverse Patchmatch for High-Resolution Optical Flow. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8925–8934, 2022. 2, 7, 17, 18

Efficient Correlation Volume Sampling for Ultra-High-Resolution Optical Flow Estimation

Supplementary Material

Block Size	Row-Major	Patch-Major	Improvement
2 1		1.6 ± 0.4	−
2 2	2.5 ± 0.5	2.3 ± 0.5	8%
2 4	6.7 ± 1.1	4.2 ± 0.7	37%
2 8	20.6 ± 4.4	8.6 ± 1.4	58%
162	28.2 ± 8.0	27.1 ± 1.2	4%

Table 4. Percentage of sampled correlation volume cells depending on the block size and layout, measured over the SINTEL [2] training dataset.

8. Method

In this section we provide more information on the sampling patterns, as well as implementation details and complexity analysis of our method.

8.1. Correlation Volume Sampling Patterns

In Table 4, we show that averaging the number of sampled values per row-major block slightly increases the ratio of cells that need to be computed but is still significantly lower than the full matrix for sufficiently small block sizes. In the second column we show that by averaging rows in our patch-major manner, sparsity is significantly increased without any computational overhead. At the largest measured block size of 16² , it becomes too large to maintain high level sparsity.

8.2. Implementation Details

We split the implementation in 3 parts according to our algorithmic design: 1) setting the computation mask, 2) performing sampled matrix-matrix multiplication, and 3) sampling the block sparse correlation volume. Our initial Py-Torch-only implementation showed significant overhead for index computations thus we implement the first and third step as small CUDA kernels. A simplified pseudocode is provided in Algorithm 1. It computes a single level whereas for multi-level pyramid an average-pooled F is used for each level.

To perform sampled matrix-matrix multiplication, in our initial version we first transformed the mask to block sparse row (BSR) format and then used the torch.sparse. triton ops.sampled addmm implementation. However, when measuring the performance we observed that up to 37% was spent on constructing the BSR matrix. To remove this overhead, we update the implementation to take blockIds directly as an input, instead

Shot	Start Frame	End Frame
010 0050	101	196
040 0040	101	264
060 0130	101	175

Table 5. List of rendered CHARGE sequences.

of BSR column offsets, and iterate over every row, while skipping the blocks with blockIds < 0. We cache the computed blocks and masks, and in every iteration compute and append blocks that were not previously computed. The previous offset is added to the block identifiers.

8.3. Memory Complexity Analysis

Here we show that our algorithm achieves at most subquadratic memory complexity in the number of pixels.

Assuming equal-size feature maps, let P = H × W be the number of feature pixels per image, B the block size, and K = 2r + 1 the lookup region size. Our algorithm requires storing one copy of the per-block mask (note that block indices can be computed in-place), taking ( P ^B² ) 2 units, and all computed blocks, with B⁴ memory units per block.

As all lookup offsets are continuous, each source pixel can request to compute at most ^K+B−¹ B 2 bins per iteration, resulting at most P ^K+B−¹ B 2 B⁴ blocks per iteration if each pixels requests a different block.

Thus by choosing B = P ⁸ , the total memory required is:

$$\left(\frac{P}{B^2}\right)^2 + P \left\lfloor \frac{K+B-1}{B} \right\rfloor^2 B^4$$

$$= \frac{P^2}{B^4} + P(1 + \left\lfloor \frac{K-1}{B} \right\rfloor)^2 B^4$$

$$= \frac{P^2}{(P^{\frac{1}{8}})^4} + P(1 + \left\lfloor \frac{K-1}{P^{\frac{1}{8}}} \right\rfloor)^2 (P^{\frac{1}{8}})^4$$

$$= P^{2-\frac{1}{8}^4} + P^{1+\frac{1}{8}^4} (1 + \left\lfloor \frac{K-1}{P^{\frac{1}{8}}} \right\rfloor)$$

$$= P^{1.5} + P^{1.5} (1 + \left\lfloor \frac{K-1}{P^{\frac{1}{8}}} \right\rfloor)$$

$$= P^{1.5} (2 + \left\lfloor \frac{K-1}{P^{\frac{1}{8}}} \right\rfloor) \in \mathcal{O}(P^{1.5}).$$ (3)

In practice, the number of requested blocks is lower due to smoothness of the optical flow and multiple source pixels requesting the same block, as shown in Table 4.

Algorithm 1 Single-Level Correlation Volume Sampling

Require: Flattened input features F ¹,² ∈ R [H×W]×D, source pixels Y ∈ R H×W×2 , lookup centroids X ∈ R N×H×W×2 , block size B, lookup radius r.

bH, bW ← ⌈H/B⌉, ⌈W/B⌉ ▷ Setting the number of blocks
M ← [0][bH×bW]×[bH×bW] ▷ Initialize computation mask as 0
for i = 0, 1, . . . , N − 1 do ▷ For every lookup iteration
  On Device for all {[y, x] ∈ [Y, Xi
                         ]}×{dx ∈ {−r, −r + 1, ..., r}
                                             2}
    M⌊y/B⌋,⌊(x+dx)/B⌋ ← 1
  End Device
  blockIds ← cumulativeSum(M) ▷ Find indices of each block
  blockIds[M != 1] ← -1 ▷ Avoid computing non-masked entries
  updateCache(blocks, blockIds, M)
  blocks ← sampledBlockSparseMMM(F
                                  1, F
                                      2, blockIds)
  On Device for all {[y, x] ∈ [Y, Xi
                         ]}
    shared memory localBlock[(2r + 2)2]
    for all dx′ ∈ {−r, −r + 1, ..., r, r + 1}
                              2 do ▷ Parallelized across threads
       blockId ← blockIds[⌊y/B⌋][
                                 (x + dx′ − r)/B
                                             ]
       tmp ← blocks[blockId][⌊y⌋ − B⌊y/B⌋][
                                         x + dx′
                                                − B
                                                    (x + dx′
                                                          )/B
       localBlock[dx′ + r] ← tmp
    end for
    for all dx ∈ {−r, −r + 1, ..., r}
                         2 do ▷ Parallelized across threads
       sample localBlock[x − B⌊(x + dx)/B⌋ + r]
    end for
  End Device
end for

9. Evaluation

In this section we provide extended evaluation results.

9.1. Correlation Sampling Measurements

In Table 6, we provide full benchmarking results, corresponding to all runtime and memory usage plots. And further analyze the impact of number of iterations, input feature dimensions, and GPU model.

9.1.1. Number of Iterations

In Figure 7, we show the impact of number of flow update iterations on the runtime and memory usage. All methods show approximately linear runtime increase.

While during the first step of our method and the default implementation a larger initial correlation volume is being computed, in practice it is not a bottleneck and they outperform on-demand sampling even with a smaller number of iterations. All methods also show near-constant peak memory consumption.

9.1.2. Feature Dimensionality

In Fig. 8, we report the results at different feature F ¹,² dimensionality. The runtime of our method and the default implementation increases sub-linearly with respect to the number of feature dimensions. This is due to the matrixmatrix multiplications not being the bottleneck. On the other hand, the runtime of on-demand sampling increases significantly with a large number of features.

As both our method and on-demand sampling stores down-sampled replicas of the target features for multi-level computations, they show increase of consumed memory with larger dimensions. However, the absolute increase is small and typically negligible.

9.1.3. Hardware

We perform additional experiments with different GPU models - NVIDIA RTX 3090, NVIDIA RTX 4090, and NVIDIA A100 - and show results in Figure 9. It can be observed that hardware has little impact on the relative performance between different methods.

9.2. Dataset Generation Details

For the data generation, we use the CYCLES renderer to generate 335 frames from 3 sequences of the BLENDER movie CHARGE as listed in Table 5. Sample frames from each of the sequences can be seen in Figures 10-12.

To improve the quality of the rendered motion vectors, we remove lens flares and volumes, and disable motion blur to decrease noise levels. Following [18], we render super-

Figure 7. Runtime and peak memory consumption by the number of flow update iterations.

Figure 8. Runtime and peak memory consumption by input feature dimensionality.

resolved motion vectors at 16K resolution with four ground truth values for each pixel. The complete Python script used to apply scene adjustments is provided in Listing 1.

9.3. Extended Results

In Table 7, we provide full results of all evaluated methods. The first part of the table contains methods that do not employ correlation volume sampling and do not benefit from performance improvements using our efficient volume sampling algorithm.

In Figures 10-12, we provide additional qualitative comparisons.

We extend PTLFlow [20] to evaluate the majority of methods using checkpoints fined tuned on the Sintel dataset, unless otherwise noted. We use correlation block size B = 8 for all results apart from MS-RAFT+, CCMR, CCMR+ where block size B = 16 is used whenever the maximum spatial dimension exceeds 1000 pixels. FlowFormer results were obtained with their default tiling technique.

9.4. Additional Datasets

In addition to the 8K dataset used for most experiments, we perform runtime measurements on high resolution Spring [18] dataset (subval split following SEA-RAFT [30]) as well as low resolution Sintel [2] and KITTI [19] datasets and report results in Tables 8 -10.

Despite these datasets having a much lower resolution than our target, making the default implementation feasible, our method often can still provide significant runtime improvements.

Since most methods are trained on the Sintel dataset and MS-RAFT+ is trained on KITTI, the error measurements are not representative. Therefore, we refer to the official benchmarks for their scores.

Figure 9. Runtime depending on input image size and different hardware.

| | | | Runtime (mean ± std), ms | | | | Peak Memory (mean ± std), MB | | | | |------------------------|-------|----------|--------------------------|------------|-------------|-----------|------------------------------|--------------|--|--| | Variable | Value | Default | On-Demand | Ours | Improvement | Default | On-Demand | Ours | | | | | 256 | 28 ± 0 | 272 ± 0 | 53 ± 0 | 80.6% | 4 ± 0 | 3 ± 0 | 8 ± 0 | | | | | 512 | 28 ± 0 | 318 ± 5 | 54 ± 1 | 82.9% | 27 ± 0 | 12 ± 0 | 34 ± 0 | | | | Input Image Width | 1024 | 29 ± 0 | 388 ± 11 | 57 ± 2 | 85.4% | 291 ± 0 | 47 ± 0 | 136 ± 2 | | | | RTX 3090 | 2048 | 80 ± 0 | 816 ± 15 | 67 ± 8 | 91.8% | 4090 ± 0 | 184 ± 0 | 588 ± 58 | | | | | 4096 | OOM | 2644 ± 48 | 302 ± 37 | 88.6% | OOM | 737 ± 0 | 2926 ± 1023 | | | | | 256 | 16 ± 0 | 197 ± 0 | 32 ± 0 | 83.9% | 4 ± 0 | 3 ± 0 | 8 ± 0 | | | | | 512 | 16 ± 1 | 233 ± 4 | 33 ± 1 | 86.0% | 27 ± 0 | 12 ± 0 | 34 ± 0 | | | | Input Image Width | 1024 | 16 ± 1 | 273 ± 10 | 35 ± 2 | 87.3% | 291 ± 0 | 47 ± 0 | 136 ± 2 | | | | RTX 4090 | 2048 | 42 ± 0 | 381 ± 12 | 42 ± 6 | 89.0% | 4090 ± 0 | 184 ± 0 | 588 ± 58 | | | | | 4096 | OOM | 1176 ± 33 | 138 ± 21 | 88.2% | OOM | 737 ± 0 | 2926 ± 1023 | | | | | 256 | 24 ± 0 | 385 ± 0 | 49 ± 0 | 87.4% | 4 ± 0 | 3 ± 0 | 8 ± 0 | | | | | 512 | 25 ± 2 | 443 ± 6 | 50 ± 5 | 88.7% | 27 ± 0 | 12 ± 0 | 34 ± 0 | | | | Input Image Width | 1024 | 26 ± 0 | 510 ± 15 | 51 ± 3 | 90.0% | 291 ± 0 | 47 ± 0 | 136 ± 2 | | | | A100 | 2048 | 60 ± 0 | 720 ± 22 | 62 ± 10 | 91.4% | 4090 ± 0 | 184 ± 0 | 588 ± 58 | | | | | 4096 | 683 ± 3 | 1932 ± 83 | 201 ± 37 | 89.6% | 71286 ± 0 | 737 ± 0 | 2926 ± 1023 | | | | | 8192 | OOM | 8011 ± 193 | 1665 ± 318 | 79.2% | OOM | 2944 ± 0 | 15384 ± 7345 | | | | | 256 | 22 ± 0 | 351 ± 0 | 41 ± 1 | 88.2% | 4 ± 0 | 3 ± 0 | 8 ± 0 | | | | | 512 | 22 ± 0 | 388 ± 6 | 41 ± 1 | 89.5% | 27 ± 1 | 12 ± 0 | 34 ± 0 | | | | Input Image Width | 1024 | 23 ± 0 | 439 ± 15 | 45 ± 2 | 89.8% | 291 ± 0 | 47 ± 0 | 136 ± 2 | | | | GH200 | 2048 | 24 ± 0 | 591 ± 20 | 48 ± 5 | 91.9% | 4091 ± 0 | 184 ± 0 | 588 ± 58 | | | | | 4096 | 191 ± 25 | 1186 ± 33 | 124 ± 34 | 89.5% | 71288 ± 0 | 737 ± 0 | 2926 ± 1023 | | | | | 8192 | OOM | 3459 ± 61 | 1004 ± 276 | 71.0% | OOM | 2944 ± 0 | 15384 ± 7345 | | | | | 1 | 8 ± 0 | 18 ± 0 | 3 ± 0 | 82.0% | 4056 ± 0 | 150 ± 0 | 535 ± 0 | | | | | 8 | 12 ± 0 | 149 ± 6 | 14 ± 2 | 90.4% | 4091 ± 0 | 184 ± 0 | 585 ± 51 | | | | Number of Iterations | 16 | 16 ± 0 | 296 ± 11 | 26 ± 3 | 91.3% | 4091 ± 0 | 184 ± 0 | 587 ± 56 | | | | GH200 | 24 | 20 ± 0 | 444 ± 16 | 38 ± 4 | 91.5% | 4091 ± 0 | 184 ± 0 | 588 ± 57 | | | | | 32 | 24 ± 0 | 591 ± 20 | 48 ± 5 | 91.9% | 4091 ± 0 | 184 ± 0 | 588 ± 58 | | | | | 64 | 24 ± 0 | 151 ± 5 | 46 ± 5 | 69.7% | 4090 ± 0 | 150 ± 0 | 472 ± 58 | | | | | 128 | 24 ± 0 | 297 ± 9 | 46 ± 5 | 84.5% | 4090 ± 0 | 161 ± 0 | 512 ± 58 | | | | Feature Dimensionality | 256 | 24 ± 0 | 591 ± 20 | 48 ± 5 | 91.9% | 4091 ± 0 | 184 ± 0 | 588 ± 58 | | | | GH200 | 512 | 27 ± 0 | 1177 ± 40 | 49 ± 5 | 95.8% | 4091 ± 0 | 243 ± 0 | 744 ± 58 | | | | | 1024 | 30 ± 2 | 2350 ± 81 | 51 ± 6 | 97.8% | 4091 ± 0 | 446 ± 0 | 1054 ± 58 | | |

Table 6. Full correlation volume sampling isolated benchmarking results depending on one variable. We report the runtime and peak memory usage of each implementation, as well as the runtime improvement over the on-demand sampling method. OOM indicates that the method fails with an out-of-memory error.

Listing 1. Blender scene setup script
import bpy
# Choose t o r e n d e r RGB or Flow pa ss
r e n d e r f l o w = F a l s e
f o r s c e n e in bpy . d a t a . s c e n e s :
      width = 1024
       a s p e c t r a t i o = width / s c e n e . r e n d e r . r e s o l u t i o n x
       h e i g h t = i n t ( round ( s c e n e . r e n d e r . r e s o l u t i o n y * a s p e c t r a t i o ) )
       s c a l e = 16 i f r e n d e r f l o w e l s e 8
      s c e n e . r e n d e r . r e s o l u t i o n x = width * s c a l e
      s c e n e . r e n d e r . r e s o l u t i o n y = h e i g h t * s c a l e
      s c e n e . r e n d e r . u s e m o t i o n b l u r = F a l s e
      s c e n e . r e n d e r . e n g i n e = "CYCLES"
      s c e n e . c y c l e s . d e v i c e = "CPU"
      n s a m p l e s = 1 i f r e n d e r f l o w e l s e 1024
      s c e n e . c y c l e s . samples = n s a m p l e s
      s c e n e . c y c l e s . a d a p t i v e m i n s a m p l e s = n s a m p l e s > 1
      s c e n e . c y c l e s . a d a p t i v e t h r e s h o l d = 0 . 0 1
      s c e n e . c y c l e s . u s e a d a p t i v e s a m p l i n g = True
      s c e n e . c y c l e s . d e n o i s e r = "OPENIMAGEDENOISE"
      s c e n e . c y c l e s . u s e d e n o i s i n g = ( not r e n d e r f l o w )
      # Removing v e r y r a r e b u t s t r o n g f i r e f l i e s
      s c e n e . c y c l e s . s a m p l e c l a m p d i r e c t = 5 0 . 0
      # Removing l e n s f l a r e s and o v e r l a y s
      s c e n e . u s e n o d e s = F a l s e
       c o l l e c t i o n = bpy . d a t a . c o l l e c t i o n s . g e t ( " f l a r e s " )
       i f c o l l e c t i o n :
             bpy . d a t a . c o l l e c t i o n s . remove ( c o l l e c t i o n )
      f o r v i e w l a y e r in s c e n e . v i e w l a y e r s :
             v i e w l a y e r . u s e p a s s c o m b i n e d = True

# Removing volumes
f o r mat in bpy . d a t a . m a t e r i a l s :
      i f mat . u s e n o d e s :
            f o r node in mat . n o d e t r e e . nodes :
                  i f node . b l s t a t i c t y p e == "OUTPUT MATERIAL" and node . i s a c t i v e o u t p u t :
                        f o r l i n k in node . i n p u t s [ " Volume " ] . l i n k s :
                              mat . n o d e t r e e . l i n k s . remove ( l i n k )

			Metrics			Runtime














































1/1	13.3	2.70	31.6	21.56	OOM	2868.3 ± 59.0	2000.8 ± 35.3

Table 7. Extended qualitative evaluation of optical flow estimation methods on a 8K dataset. We split the results depending on the evaluation scale and report endpoint-error (EPE), 1px outlier rate, as well as metrics for pixels with large motion (magnitude over 128px). Best result of each metric is highlighted in bold, best performing scale for each method is underlined. OOM indicates that the method fails with an out-of-memory error, while n/a - not applicable.

	1px error % ↓	EPE px ↓	Best Runtime, s	Without Ours, s	Our Improvement
GMFlow [32]	3.6	0.37	0.97		n/a
PWC-Net [25]	1.1	0.19	0.09		n/a
Flow-1D [31]	1.5	0.17	0.25		n/a
DIP [39]	0.6	0.15	1.00		n/a
FlowFormer [8]	0.6	0.13	1.83		n/a
FlowFormer++ [22]	0.6	0.13	1.84		n/a
SCV [13]	2.1	0.18	3.40		n/a
HCVFlow [38]	0.7	0.13	0.34		n/a
MS-RAFT+ [11]	0.5	0.11	1.62	1.69	−4%
RAFT [28]	0.7	0.15	0.33	0.33	+0%
CCMR [10]	0.7	0.12	1.31	1.60	−18%
CCMR+ [10]	0.5	0.14	4.09	4.25	−4%
SEA-RAFT [30]	0.5	0.12	0.13	0.17	−23%

Table 8. Quantitative evaluation on Spring validation split.

	1px error % ↓	EPE px ↓	Best Runtime, s	Without Ours, s	Our Improvement
GMFlow [32]	11.4	1.11	0.10		n/a
PWC-Net [25]	16.4	2.34	0.04		n/a
Flow-1D [31]	15.4	2.16	0.08		n/a
DIP [39]	11.9	1.49	0.29		n/a
FlowFormer [8]	8.5	0.61	0.81		n/a
FlowFormer++ [22]	8.4	0.60	0.82		n/a
SCV [13]	11.6	2.18	0.94		n/a
HCVFlow [38]	11.0	1.11	0.11		n/a
MS-RAFT+ [11]	8.7	1.11	0.36	0.46	−23%
RAFT [28]	11.4	1.22	0.08	0.08	+0%
CCMR [10]	8.6	0.79	0.33	0.33	+0%
CCMR+ [10]	7.5	0.74	0.87	1.01	−15%
SEA-RAFT [30]	6.1	0.58	0.03	0.03	+0%

Table 9. Quantitative evaluation on Sintel training dataset.

	1px error % ↓	EPE px ↓	Best Runtime, s	Without Ours, s	Our Improvement
GMFlow [32]	25.6	1.97	0.10		n/a
PWC-Net [25]	30.4	3.12	0.05		n/a
Flow-1D [31]	37.8	4.48	0.08		n/a
DIP [39]	27.5	3.96	0.30		n/a
FlowFormer [8]	19.4	1.09	0.80		n/a
FlowFormer++ [22]	19.1	1.08	0.81		n/a
SCV [13]	24.0	3.35	0.91		n/a
HCVFlow [38]	23.2	1.46	0.11		n/a
MS-RAFT+ [11]	12.3	0.68	0.38	0.48	−21%
RAFT [28]	24.0	1.55	0.08	0.08	+0%
CCMR [10]	18.4	1.09	0.34	0.34	+0%
CCMR+ [10]	16.7	1.02	0.93	1.06	−12%
SEA-RAFT [30]	13.8	0.88	0.03	0.03	+0%

Table 10. Quantitative evaluation on KITTI 2015 training dataset.

Figure 10. Qualitative comparison across different evaluation scales on a frame from 010 0050 shot. Image from Charge by Blender Studio.

Figure 11. Qualitative comparison across different evaluation scales on a frame from 040 0040 shot. Image from Charge by Blender Studio.

Figure 12. Qualitative comparison across different evaluation scales on a frame from 060 0130 shot. Image from Charge by Blender Studio.

briedisefficientcorrelationvolumesampling2025.md 257 KB 文件历史 原始文件

status: converted

Efficient Correlation Volume Sampling for Ultra-High-Resolution Optical Flow Estimation

Abstract

1. Introduction

2. Related Work

3. Correlation Volume Sampling Analysis

3.1. Problem and Baseline Implementation

3.2. Correlation Volume Access Patterns

4. Efficient Correlation Volume Sampler

4.1. Input Preprocessing

4.2. Setting Computation Mask

4.3. Sparse Correlation Volume Computation

4.4. Sparse Volume Sampling

4.5. Caching Across Iterations

5. Isolated Sampling Evaluation

5.1. Isolated Correlation Sampling Results

5.1.1. Image Resolution

5.1.2. Other variables

5.2. Component Analysis

6. Ultra-High-Resolution Evaluation

6.1. Dataset

6.2. Cascaded Inference

6.3. Evaluation Results

7. Conclusion

References

Efficient Correlation Volume Sampling for Ultra-High-Resolution Optical Flow Estimation

Supplementary Material

8. Method

8.1. Correlation Volume Sampling Patterns

8.2. Implementation Details

8.3. Memory Complexity Analysis

Algorithm 1 Single-Level Correlation Volume Sampling

9. Evaluation

9.1. Correlation Sampling Measurements

9.1.1. Number of Iterations

9.1.2. Feature Dimensionality

9.1.3. Hardware

9.2. Dataset Generation Details

9.3. Extended Results

9.4. Additional Datasets

briedisefficientcorrelationvolumesampling2025.md 257 KB

文件历史原始文件