Abstract: The benefits of medical imaging are enormous. Medical images provide considerable amounts of anatomical information and this facilitates medical practitioners in performing effective disease diagnosis and deciding upon the best course of medical treatment. A transition from traditional monochromatic medical images like CT scans, X-Rays or MRI images to a colored 3D representation of the anatomical structure further enhances the capabilities of medical professionals in extracting valuable medical information. The proposed framework in our research starts with performing color transfer by finding deep semantic correspondence between two medical images: a colored reference image, and a monochromatic CT scan or an MRI image. We extend this idea of reference-based colorization technique to perform colored volume rendering from a stack of grayscale medical images. Furthermore, we also propose to use an effective reference image recommendation system to aid in the selection of good reference images. With our approach, we successfully perform colored medical volume visualization and essentially eliminate the painstaking process of user interaction with a transfer function to obtain color and opacity parameters for volume rendering.