Non-invasive and accurate methods are needed to identify patients with clinically significant portal hypertension (CSPH). We investigated the ability of deep convolutional neural network (CNN) analysis of computed tomography (CT) or magnetic resonance (MR) to identify patients with CSPH.We collected liver and spleen image from patients who underwent contrast-enhanced CT or MR analysis within 14 days of transjugular catheterization for hepatic venous pressure gradient measurement. The CT cohort comprised participants with cirrhosis in the CHESS1701 study, performed at 4 university hospitals in China from August 2016 through September 2017. The MR cohort comprised participants with cirrhosis in the CHESS1802 study, performed at 8 university hospitals in China and 1 in Turkey from December 2018 through April 2019. Patients with CSPH were identified as those with a hepatic venous pressure gradient =10 mmHg. In total, we analyzed 10014 liver images and 899 spleen images collected from 679 participants who underwent CT analysis, and 45554 liver and spleen images from 271 participants who underwent MR analysis. For each cohort, participants were shuffled and then randomly and equiprobably sampled for 6 times into training, validation, and test datasets (ratio of 3:1:1). Therefore, a total of 6 deep CNN models for each cohort were developed for identification of CSPH.The CT-based CNN analysis identified patients with CSPH with the area under receiver operating characteristic curve (AUC) value of 0.998 in the training set (95% CI, 0.996-1.000), an AUC of 0.912 in the validation set (95% CI, 0.854-0.971), and an AUC of 0.933 (95% CI, 0.883-0.984) in the test datasets. The MR-based CNN analysis identified patients with CSPH with an AUC of 1.000 in the training set (95% CI, 0.999-1.000), and AUC of 0.924 in the validation set (95% CI, 0.833-1.000), and an AUC of 0.940 in the test dataset (95% CI, 0.880-0.999). When the model development procedures were repeated 6 times, and AUCs for all CNN analyses were 0.888 or greater, with no significant differences between rounds (P>.05).We developed a deep CNN to analyze CT or MR images of liver and spleen from patients with cirrhosis that identifies patients with CSPH with an AUC value of 0.9. This provides a non-invasive and rapid method for detection of CSPH (ClincialTrials.gov number, NCT03138915; NCT03766880).