Immunotherapy is an effective treatment for esophageal cancer (ESCA) patients. However, there are no dependable markers for predicting prognosis and immunotherapy responses in ESCA. Our study aims to explore immune gene prognostic models and markers in ESCA as well as predictors for immunotherapy. The expression profiles of ESCA were obtained from The Cancer Genome Atlas (TCGA), the Gene Expression Omnibus (GEO), and International Cancer Genome Consortium (ICGC) databases. Cox regression analysis was performed to construct an immune gene prognostic model. ESCA was grouped into three immune cell infiltration (ICI) clusters by CIBERSORT algorithm. The immunotherapy response of patients in different ICI score clusters was also compared. The copy number variations, somatic mutations, and single nucleotide polymorphisms were analyzed. Enrichment analyses were also performed. An immune gene prognostic model was successfully constructed. The ICI score may be used as a predictor independent of tumor mutation burden. Enrichment analyses showed that the differentially expressed genes were mostly enriched in microvillus and the KRAS and IL6/JAK/STAT3 pathways. The top eight genes with the highest mutation frequencies in ESCA were identified and all related to the prognosis of ESCA patients. Our study established an effective immune gene prognostic model and identified markers for predicting the prognosis and immunotherapy response of ESCA patients.