Skin cancer is one of the most prevalent types of cancer, has been rising quickly in recent decades. Smoking, Ultraviolet rays, DNA changes and poor lifestyle choices are the main contributors of skin cancer. This paper presented a novel deep learning-based GOA-CRA method is used for sequentially identifying images in the skin cancer. Initially, the images are gathered from the ISIC dataset and undergo multi-scale retinex pre-processing (MSR) to improve the quality of the images. Then, the pre-processed images are used as input for the Grasshopper Optimization Algorithm (GOA) for removing the hairs overlapping in the area of skin. The segmented hair region with the mask and the pre-processed images are taken as an input to the Contextual Residual Aggregation (CRA) network for recreating the image by inpainting the missing pixels. Finally, the proposed GOA-CRA method is used to classify the skin cancer as melanoma skin cancer or not. The proposed GOA-CRA improves the overall accuracy of 5.42%, 4.21%, 5.06% and 2.93% better than U-Net, Link Net, Dense Net, Res Net respectively