Merge branch 'smc-fix'

D. Wythe says:

====================
fix unexpected SMC_CLC_DECL_ERR_REGRMB error

We can easily trigger the SMC_CLC_DECL_ERR_REGRMB exception within
following script:

server: smc_run nginx
client: smc_run  ./wrk -c 2000 -t 8 -d 20 http://smc-server

And we can clearly see that this error is also divided into two types:

1. 0x09990003
2. 0x05000000/0x09990003

Which has the same root causes, but the immediate causes vary.

The root cause of this issues is that remove connections from link group
is not synchronous with add/delete rtoken entry,  which means that even
the number of connections is less that SMC_RMBS_PER_LGR_MAX, it does not
mean that the connection can register rtoken successfully later. In
other words, the rtoken entry may released, This will cause an
unexpected SMC_CLC_DECL_ERR_REGRMB to be reported, and then this SMC
connections have to fallback to TCP.

This patch set handles two types of SMC_CLC_DECL_ERR_REGRMB exceptions
from different perspectives.

Patch 1: fix the 0x05000000/0x09990003 error.
Patch 2: fix the 0x09990003 error.

After those patches, there is no SMC_CLC_DECL_ERR_REGRMB exceptions in
my
test case any more.

v1 -> v2:
- add bugfix patch for SMC_CLC_DECL_ERR_REGRMB cause by server side
v2 -> v3:
- fix incorrect mail thread
====================

Signed-off-by: David S. Miller <davem@davemloft.net>

net/smc/smc_core.c

@@ -1161,8 +1161,8 @@ void smc_conn_free(struct smc_connection *conn)
 			cancel_work_sync(&conn->abort_work);
 	}
 	if (!list_empty(&lgr->list)) {
-		smc_lgr_unregister_conn(conn);
 		smc_buf_unuse(conn, lgr); /* allow buffer reuse */
+		smc_lgr_unregister_conn(conn);
 	}
 
 	if (!lgr->conns_num)
@@ -1864,7 +1864,8 @@ int smc_conn_create(struct smc_sock *smc, struct smc_init_info *ini)
 		    (ini->smcd_version == SMC_V2 ||
 		     lgr->vlan_id == ini->vlan_id) &&
 		    (role == SMC_CLNT || ini->is_smcd ||
-		     lgr->conns_num < SMC_RMBS_PER_LGR_MAX)) {
+		    (lgr->conns_num < SMC_RMBS_PER_LGR_MAX &&
+		      !bitmap_full(lgr->rtokens_used_mask, SMC_RMBS_PER_LGR_MAX)))) {
 			/* link group found */
 			ini->first_contact_local = 0;
 			conn->lgr = lgr;