Skip to content

Commit

Permalink
drm/amdgpu: Don't pin VRAM without DMABUF_MOVE_NOTIFY
Browse files Browse the repository at this point in the history
Pinning of VRAM is for peer devices that don't support dynamic attachment
and move notifiers. But it requires that all such peer devices are able to
access VRAM via PCIe P2P. Any device without P2P access requires migration
to GTT, which fails if the memory is already pinned for another peer
device.

Sharing between GPUs should not require pinning in VRAM. However, if
DMABUF_MOVE_NOTIFY is disabled in the kernel build, even DMABufs shared
between GPUs must be pinned, which can lead to failures and functional
regressions on systems where some peer GPUs are not P2P accessible.

Disable VRAM pinning if move notifiers are disabled in the kernel build
to fix regressions when sharing BOs between GPUs.

Signed-off-by: Felix Kuehling <felix.kuehling@amd.com>
Tested-by: Hao (Claire) Zhou <hao.zhou@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
(cherry picked from commit 0518581)
  • Loading branch information
Felix Kuehling authored and Alex Deucher committed Apr 22, 2025
1 parent 5cf3c60 commit 5e56935
Showing 1 changed file with 12 additions and 5 deletions.
17 changes: 12 additions & 5 deletions drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c
Original file line number Diff line number Diff line change
Expand Up @@ -81,14 +81,21 @@ static int amdgpu_dma_buf_pin(struct dma_buf_attachment *attach)

dma_resv_assert_held(dmabuf->resv);

/*
* Try pinning into VRAM to allow P2P with RDMA NICs without ODP
/* Try pinning into VRAM to allow P2P with RDMA NICs without ODP
* support if all attachments can do P2P. If any attachment can't do
* P2P just pin into GTT instead.
*
* To avoid with conflicting pinnings between GPUs and RDMA when move
* notifiers are disabled, only allow pinning in VRAM when move
* notiers are enabled.
*/
list_for_each_entry(attach, &dmabuf->attachments, node)
if (!attach->peer2peer)
domains &= ~AMDGPU_GEM_DOMAIN_VRAM;
if (!IS_ENABLED(CONFIG_DMABUF_MOVE_NOTIFY)) {
domains &= ~AMDGPU_GEM_DOMAIN_VRAM;
} else {
list_for_each_entry(attach, &dmabuf->attachments, node)
if (!attach->peer2peer)
domains &= ~AMDGPU_GEM_DOMAIN_VRAM;
}

if (domains & AMDGPU_GEM_DOMAIN_VRAM)
bo->flags |= AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED;
Expand Down

0 comments on commit 5e56935

Please sign in to comment.