realtek: rtl838x: fix broadcast flooding with many multicast entries

When many multicast entries are installed broadcast flooding might
potentially stop working for several ports. This is because the layer
2 broadcast flood port mask index has the wrong offset. It should be
9 bits, matching the 2^9 = 512 indexes on rtl838x, not 12.

The wrong offset leads to L2_BC_FLD_PMSK being set to 504, not 511
((511 << 12) >> 9) & 511 = 504). So, as by default an unset PMSK
is set to all ports, the issue would only become noticeable once
many multicast entries are installed, causing the 504th entry to be set
to something other than all ports.

Fixing this by setting the offset to 9 bits, to correctly point to our
511th reserved entry for all ports.

Tested-on: ZyXEL GS1900-24HP v1

Fixes: 28e972b2ea2f ("realtek: Configure initial L2 learning setup")
Signed-off-by: Linus Lüssing <[email protected]>
Link: https://github.com/openwrt/openwrt/pull/18733
Signed-off-by: Robert Marko <[email protected]>

target/linux/realtek/files-6.6/drivers/net/dsa/rtl83xx/rtl838x.c

@@ -502,7 +502,7 @@ static void rtl838x_l2_learning_setup(void)
 	/* Set portmask for broadcast traffic and unknown unicast address flooding
 	 * to the reserved entry in the portmask table used also for
 	 * multicast flooding */
-	sw_w32(UNKNOWN_MC_PMASK << 12 | UNKNOWN_MC_PMASK, RTL838X_L2_FLD_PMSK);
+	sw_w32(UNKNOWN_MC_PMASK << 9 | UNKNOWN_MC_PMASK, RTL838X_L2_FLD_PMSK);
 
 	/* Enable learning constraint system-wide (bit 0), per-port (bit 1)
 	 * and per vlan (bit 2) */