It's possible it could be nothing more that just some local weirdo. Stealing panties doesn't trigger or always precipitate a sexual assault. Makes notes of anything you can remember about that guy and his car. Make, model, color, license plate number, etc. Let hubby know what happened and maybe don't wash clothes by yourself for a bit.
You can always call the local law enforcement agency and speak with someone. Let 'em know what happened and what info you have to give. Often times people don't call at all, or they call and don't have specific info besides "he was acting weird."
Above all, don't be paranoid, just be "aware."
Of every one hundred men, ten should not even be here. Eighty are nothing but targets. Nine are real fighters, and we are lucky to have them, for they make the battle. Ah, but the One... One of them is a Warrior... He will bring the others back.
"Wrong door, buddy!"
Let no man's ghost say my training failed him.