Skip to content

Commit

Permalink
io_uring: defer iowq cqe overflow via task_work
Browse files Browse the repository at this point in the history
Don't handle CQE overflows in io_req_complete_post() and defer it to
flush_completions. It cuts some duplication, and I also want to limit
the number of places directly overflowing completions.

Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/9046410ac27e18f2baa6f7cdb363ec921cbc3b79.1742829388.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
  • Loading branch information
Pavel Begunkov authored and Jens Axboe committed Mar 25, 2025
1 parent 3f0cb8d commit 3afcb3b
Showing 1 changed file with 7 additions and 4 deletions.
11 changes: 7 additions & 4 deletions io_uring/io_uring.c
Original file line number Diff line number Diff line change
Expand Up @@ -892,6 +892,7 @@ bool io_req_post_cqe(struct io_kiocb *req, s32 res, u32 cflags)
static void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags)
{
struct io_ring_ctx *ctx = req->ctx;
bool completed = true;

/*
* All execution paths but io-wq use the deferred completions by
Expand All @@ -905,18 +906,20 @@ static void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags)
* the submitter task context, IOPOLL protects with uring_lock.
*/
if (ctx->lockless_cq || (req->flags & REQ_F_REISSUE)) {
defer_complete:
req->io_task_work.func = io_req_task_complete;
io_req_task_work_add(req);
return;
}

io_cq_lock(ctx);
if (!(req->flags & REQ_F_CQE_SKIP)) {
if (!io_fill_cqe_req(ctx, req))
io_req_cqe_overflow(req);
}
if (!(req->flags & REQ_F_CQE_SKIP))
completed = io_fill_cqe_req(ctx, req);
io_cq_unlock_post(ctx);

if (!completed)
goto defer_complete;

/*
* We don't free the request here because we know it's called from
* io-wq only, which holds a reference, so it cannot be the last put.
Expand Down

0 comments on commit 3afcb3b

Please sign in to comment.