I recently had the exact same interaction, but I asked the guy his name and if his property operated a business. I then pulled out my phone and made it clear I was calling the cops. I told him I needed his info as I needed to know who I was going to have charged and sued for violating my charter given rights.
He stated we didn't need to do that but I still had to leave,
I responded with he has 3 options,
1 me calling the cops
2 me posting on all my social media where the best brown trout fishing is on this stream and he'll have to deal with all of us legally fishing
3 he can walk back to his house educated
I also told him at the end of this id still be fishing it. He walked away muttering. We have to stand up to these guys. I have talked to anglers that have been scared off of this property numerous time. The land owner does it because he gets away with it.
I also go to local detachment if the land owner gets threatening, even if nothing happens just to have the record.
Stand for our rights guys, we loose stuff daily.