Can we use custom xslt in BizTalk map?

Yes ,  There are two options   available for that –

1. You can  choose  scripting functoids.
2. Custom XSLT path of  map grid   property.