Butturbur's Worst Customers

[Inziladun]

Some points raised in this thread recently regarding whether the wights could leave the Barrow-downs got me wondering about something.
Aragron makes this statement to the Council of Elrond, mainly for Boromir's edification:

'Peace and freedom, do you say? The North would have known them little but for us. Fear would have destroyed them.....And yet less thanks have we than you..."Strider" I am to one fat man who lives within a day's march of foes that would freeze his heart, or lay his little town in ruin, if he were not guarded ceaselessly.'

The only creatures of evil intent that near to Bree would seem to be the wights. I could imagine them indeed 'freezing Butturbur's heart', or driving everyone from Bree. But the wights don't seem to have the inclination, or the capability, to do so. Aragorn's words suggest that the instant the Dúnedain let their guard down, these enemies would wreak havoc in Eriador. And when the Dúnedain left to fight in the War of the Ring, enemies indeed came to Bree. But those were folk out of Dunland and Isengard, hardly 'a day's march' from Bree. The wights apparently remained right where they were, though the Dúnedain were gone.
Any thoughts on who the Dúnedain were primarily fighting off from Bree and the Shire in the Third Age?