This afternoon I ran into an issue where Chinese output from Ant in the console was fine, but the Chinese output from mxmlc was garbled. The cause was that flexTasks was using ${FLEX\_HOME}/ant/lib/flexTasks.jar from a different version of the SDK — switching to the matching one fixed it.
For the mxmlc parameter documentation of the Ant tasks provided by the FLEX SDK, see http://livedocs.adobe.com/flex/3/html/help.html?content=compilers_13.html
If you run Ant tasks in FDT and hit “Java heap space” while mxmlc is compiling a swf, the reason is that some of the swc files used as libraries are too large, or there are too many of them, so memory runs out and you need to give it more: in FDT go to window->preferences->java->installed jres-> select the current entry and click edit, then set the default vm arguments, for example -Xms64m -Xmx512m.