The HDMI 2.1 port won't do 4k120 on Linux for the reasons everyone else said above (HDMI licensing)
Gitlab discussion link
As you mentioned, you'll have to use a DIsplayPort->HDMI 2.1 adapter.
I have used both the cable matters
#102101 and
#201362-GRY successfully for 4K120 under Kubuntu 23.10 via Displayport->HDMI 2.1 on RX6600. These adapters are available on Amazon for ~$35 .
No freesync support with DP->HDMI2.1 unfortunately.
Kubuntu specific issues (might not apply to you)
SDDM sometimes doesn't like 4k120 so you have to force the SDDM res down to 1920x1080 for the login screen to appear (otherwise black screen)
My TV (Sony X90J) sometimes doesn't like 4k120@10bit via Displayport->HDMI2.1, KDE uses 10bit by default so I have to put KWIN_DRM_PREFER_COLOR_DEPTH=24 into /etc/environment to make it use 8 bit RGB. (otherwise black screen)
In general be prepared to do some troubleshooting and getting a black screen doesn't always mean it isn't going to work.