Saruman appearing to the Three Walkers

[Kalimac]

Hello everyone, I'm really new (this is only my 3rd post!) so I hope this isn't one of those topics which has been beaten to death elsewhere. It's a pretty silly question but I'm curious.

In "The Two Towers" just after Aragorn, Gimli and Legolas encounter Eomer and are camping out in under the eaves of Fangorn, they see an old man very close to them who is wearing a hat and cloak and who vanishes without a trace when they try to speak to him; they decide that it must have been Saruman (wandering around and spying) and later when they meet up with Gandalf he says that it wasn't him, so it was probably Saruman. But later on when they're all at Orthanc, in the chapter "The Voice of Saruman" he gives no indication of knowing who the Three Walkers are or what they're after, so it seems rather pointless. If it really was Saruman, wouldn't he have tried a little harder to thwart them (doing more than just scaring off the horses) or at least indicated later that he had seen them? The whole incident is scary but seems rather pointless, unless I'm missing something.

So why didn't Saruman do more to them at the time, or at least indicate later that he knew them? Also, if he's able to disapparate like that, why didn't he just do that when the Ents stormed Isengard instead of just getting trapped in Orthanc? If you have any thoughts on any of these questions I'd really like to hear them. Thanks!