Filenames

The only two invalid characters for OS X filesystems (UFS, HFS+, and HFSX) are slash ('/') and null ('\0')

Text encoding of filenames

OS X supports international unicode characters in filenames, the filename must be normalized to Apple's "nearly" Unicode NFD (NFD with Apple HFS+ variations).

OS X always uses NFD on its hfs+ filesystem (or even when using FAT on a memory stick).

Windows XP and Linux do not perform any normalization, they just store the filename as given.

(The above assumes you are using the POSIX API, not the legacy pre-OS X API.)

Related:

jroller.com - unicode file names
unicode.org - Normalization Form D (NFD)
OS X Syntax