The 3D genome is characterized by a complex organization made of genomic and epigenomic layers with profound implications on gene regulation and cell function. However, the understanding of the fundamental mechanisms driving the crosstalk between nuclear architecture and (epi)genomic information is still lacking. The plant Arabidopsis thaliana is a powerful model organism to address these questions owing to its compact genome for which we have a rich collection of microscopy, chromosome conformation capture (Hi-C) and ChIP-seq experiments. Using polymer modelling, we investigate the roles of nucleolus formation and epigenomics-driven interactions in shaping the 3D genome of A. thaliana. By validation of several predictions with published data, we demonstrate that self-attracting nucleolar organizing regions and repulsive constitutive heterochromatin are major mechanisms to regulate the organization of chromosomes. Simulations also suggest that interphase chromosomes maintain a partial structural memory of the V-shapes, typical of (sub)metacentric chromosomes in anaphase. Additionally, self-attraction between facultative heterochromatin regions facilitates the formation of Polycomb bodies hosting H3K27me3-enriched gene-clusters. Since nucleolus and heterochromatin are highly-conserved in eukaryotic cells, our findings pave the way for a comprehensive characterization of the generic principles that are likely to shape and regulate the 3D genome in many species.