We had something like this in the first couple of years of ownership of our Classic 35 (~17 years ago). The issue turned out to be due to “crud” that had lodged itself in the shaft bearing. We manage to free it/clear it by flooding the top bearing with PTFE lubricant. It took a few bottle fulls and several days. After that, we found that there should have been a plastic cap to prevent stuff from accumulating from the top (where the emergency tiller fits). We bought a replacement from Dufour (very expensive!), and it has stayed in place ever since. Our rudder has been very easy ever since.
Maybe you can try this before dropping the rudder.