Virtual reality (VR)- and augmented reality (AR)-based simulations are key technologies in Industry 4.0 which allow for testing and studying of new processes before their deployment. A simulator of industrial processes needs a flexible way in which to model the activities performed by the worker and other elements involved, such as robots and machinery. This work proposes a framework to model industrial processes for VR and AR simulators. The desk method was used to review previous research and extract the most important features of current approaches. Novel features include interaction among human workers and a variety of automation systems, such as collaborative robots, a broader set of tasks (including assembly and disassembly of components), flexibility of modeling industrial processes for different domains and purposes, a clear separation of process definition and simulator, and independence of specific programming languages or technologies. Three industrial scenarios modeled with this framework are presented: an aircraft assembly scenario, a guidance tool for high-voltage cell security, and an application for the training of machine-tool usage.