see:
http://msdn.microsoft.com/en-us/library/cc645032%28VS.95%29.aspx
You may need crossdomain.xml as well as clientaccesspolicy.xml.
e.g. crossdomain.xml
<?xml version="1.0"?>
<!DOCTYPE cross-domain-policy SYSTEM "http://www.macromedia.com/xml/dtds/cross-domain-policy.dtd">
<cross-domain-policy>
<allow-http-request-headers-from domain="*" headers="*"/>
</cross-domain-policy>
and clientaccesspolicy.xml
<?xml version="1.0" encoding="utf-8"?>
<access-policy>
<cross-domain-access>
<policy>
<allow-from http-request-headers="*">
<domain uri="*"/>
</allow-from>
<grant-to>
<resource path="/" include-subpaths="true"/>
</grant-to>
</policy>
</cross-domain-access>
</access-policy>
Also check this link:
http://timheuer.com/blog/archive/2008/04/06/silverlight-cross-domain-policy-file-snippet-intellisense.aspx
Double check that the needed files are in the correct path. I once had this problem with a flash-application which broke all the time just because the xml-file was in the wrong directory.