Indian women have worn sarees for centuries and have a significant market share in their clothing. Despite the ability to identify saree materials with tactile texture and the possibility to purchase online, it may not always be practicable to touch and feel a saree being worn. Moreover, the wide variety of saree materials available in comparison to other types of clothing, as well as the lack of experience in recognizing them, make this challenge even more challenging. The rapid advancement of smartphone technology makes it possible to capture images of cloth to determine the exact material of a saree and make an online purchase. To overcome the aforementioned challenges, a novel deep learning-based saree texture classification framework has been proposed for the rapid classification of saree tactile textures. Deep learning-based segmentation is used to detect visual texture and identify sarees from collected images. For better understanding the introduction of Mask Region-based Convolutional Neural Network (Mask RCNN) was used for generating the saree patches. To categorize saree textures, we propose a deep learning-based VGG-16 network. According to the findings, this pipeline performs better than the current deep learning methods and achieves 97.41% accuracy.