From http://en.memory-alpha.org/wiki/Headgear:
Vedek Winn’s hat was designed by Robert Blackman. He created it as an “interesting and curious shape that viewers haven’t seen. It’s about rhythm and other things.” He also agreed that the style resembles the Sydney Opera House. (Star Trek: Deep Space Nine Companion)