Clinical AI tools hold the promise to transform the practice of medicine, but lack of transparency in some tools is an ongoing challenge. One way to demystify black-box AI reasoning is the use of heat maps that ‘explain’ how an AI tool made a diagnosis based on a specific clinical image. A new study suggests that heat maps underperform human clinicians, are not ready for prime time and should be refined before wider adoption in clinical practice.