COVID-19 is a viral respiratory illness that causes characteristic lung changes on chest X-ray (CXR) and CT such as patchy opacities and diffuse inflammation. Imaging features of COVID-19, such as ground-glass opacities, are non-specific and can overlap with other lung infections, making it difficult to confirm the disease based on imaging alone. To overcome these challenges, a novel DL-based COVNET approach is proposed for COVID-19 severity detection using multi-modality images (CXR and CT). Initially, the gathered multi-modality images are pre-processed by Adaptive Wienmed (ADW) filter for denoising the input images. The noise-free images are fed into the Dual-stream EfficientNet (DSE-Net) for extracting the structural and textural features. That extracted structural and textural features are fused and fed into Deep Belief Network (DBN) for classifying the COVID-19 cases such as normal and COVID-19. The proposed COVNET model is evaluated based on its f1 score (F1), specificity (SP), precision (PR), recall (RE) and accuracy (AC). The classification AC of 99.14% for the proposed DBN are highly reliable for publicly available dataset. The proposed COVNET model achieves the overall AC by 2.82%, 5.14%, and 1.46% comparing to the existing method such as MIn-V3, CoroDet and nCOVnet respectively.