Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Implementation of an IoT-Based Environment to Control an Industrial Process by Voice Commands using a Virtual Assistant

Research Group on Artificial Intelligence and Assistive Technologies – GIIATa, Universidad Politécnica Salesiana, Cuenca-Ecuador ↗
Publication year
Digital Object Identifier10.54941/ahfe1001629

Abstract

In recent years the use of virtual assistants has been extended in a large number of applications to provide comfort to the user. This article proposes the development of an IoT (Internet of Things) environment to control an industrial process through voice commands using these devices, where the operator can interact and control the process using his voice. In the first part, a review of works where virtual assistants are used for automation and control processes is made, then the proposed architecture is presented, where we detail the protocols and media used. Next, the development of the system and the validations carried out in an industrial environment are presented, considering its own characteristics. Finally, we detail the conclusions of the project focused on minimizing access times to a control interface, and the growing use of these technologies at different levels of the automation pyramid considering its benefits and limitations

INTRODUCTION

In the area of industrial process control, human-machine interfaces allow interaction with computers through different platforms and media. The use of well-designed interfaces allows operators to work in a faster and simplified way (Calero, 2021). Speech recognition and interfaces have evolved signi- ficantly and the change in this sector has been very large (García, 2011). Moreover, new internet-based technologies provide digital interfaces to faci- litate human-machine interaction (Manrique, 2018). In the midst of the IoT boom, equipment and products connected to the Internet improve the consu- mer experience (Venegas, 2018). Thus, this project proposes the development of a voice interface through a virtual assistant connected to a Siemens PLC using a Raspberry Pi. The development seeks to reduce interaction times for operators navigating through different menus and submenus using touch screens or button panels, instead they will only have to pronounce a specific command and the system will respond in a timely manner resulting in greater

© 2022. Published by AHFE Open Access. All rights reserved. 145

146 Guachun-Arias and Serpa-Andrade

efficiency and user satisfaction. In this sense, we consider that not all auto- matization processes can be controlled with this type of interfaces and this will depend largely on the environment and the type of process variable to be controlled; in such cases the interfaces using HMI are the most appropriate option.

Currently, there are several commercial voice assistant platforms avai- lable for this purpose; companies such as Amazon, Google and Apple, with their Alexa, OK Google and Siri technologies, respectively, stand out (Diamantaras, 2019).

For our project we use Alexa, a voice service located in the Amazon Web Services cloud, whose versatility of integration makes it ideal for this type of applications (Samper and Campos).

RELATED WORKS The control of industrial automation processes, in its different lines, is gene- rally per-formed through the use of touch screens or pulsed control (Ferreira, Hernandez and Suarez, 2020). Globally, the control of an industrial process seeks that the operator / user has control of it in a safe and timely manner, including events outside the nature of the industrial environment. All these considerations have allowed industrial automation to bring a series of bene- fits both in pro-duction times and operating costs (Balaguera, 2021). On the other hand, new digital monitoring solutions integrating hardware and sof- tware allow determining and configuring states and variables in real time (Carrera, 2021). Thus, the results are reflected in different sectors of the eco- nomy, from food to telecommunications. In this sense, a brief review of the work done in this area is made.

There are works where virtual assistants have been used in automation processes. Thus, in (Chalán, 2020) a virtual assistant is used for pressure control for liquid recirculation where the process control is performed in real time, indicating start-up and circulation levels; additionally, the system can be included as part of a SCADA system for integral supervision.

On the other hand, in the work done in (Álvarez, 2020) in which the design of an optimal control through IOT tools using a Raspberry is pro- posed, where the author interacts this card with a PLC, and thus access to memory addresses to extract and place information of variables related to the process. The exposed results show the versatility of using development boards for the control of sophisticated control processes.

Additionally, the work done in (Calvopiña, Tapia and Tello, 2020) the authors propose the use of a voice assistant for climate control in a demotics’ system, where new requests and attempts (skills) are developed that allow the operator to request information to the system and this in turn, is able to respond in a timely manner, as a complement, the authors propose the use of new libraries so that the system can support different languages.

Another interesting work is presented in (Zakharyan, 2018), here the auth- ors propose the development of applications for collaborative robots. One of these applications has the function of con-trolling these robots by voice com- mands and transmitting this instruction wirelessly to the robot control unit.

Implementation of an IoT-Based Environment 147

Figure 1: General architecture of the proposed system.
Original PDF Figure 1:

The authors conclude that the implementation of commands in industrial processes can improve the performance of companies, however, they point out that the presence of noise can be detrimental to this type of automation.

METHODOLOGY

This paper proposes to implement an IOT-based environment to control an industrial process by voice commands using a virtual assistant. This section is used to describe the general architecture of the system and from there, each element of the system and its function within the system is described.

Project Architeture

Figure 1 shows the general architecture of the system, where physical ele- ments, communication protocols and electrical signals for the PLC are shown. The Raspberry development board will be the link between the elements of the environment.

The development of the environment is divided into three parts properly identified, the first one corresponds to the initial configuration of the Rasp- berry card, for this it is required to install a Raspbian operating system. The installation of Raspbian is done using the NOOBS (New out of the box softw- are) installation wizard. Addition-ally, for Raspberry to communicate with the PLC, the Modbus TCP/IP protocol is used, using a proprietary library for Phyton called Pymodbus. On the other hand, for Raspberry to be able to exchange information with the internet through the voice assistant it is neces- sary to create a secure tunnel, for our project we have used Ngrok. Finally, the communication between the Raspberry board and the PLC is done using Node-network, which is a programming tool used to connect different har- dware devices, APIs and internet services (Chuquimarca, Asencio, Torres, Bustos, Sánchez and Saldaña, 2022).

The second stage corresponds to the writing of Skills for Alexa with which requests are made to the voice assistants. For this task we must cre- ate an Amazon developer account. This platform is quite intuitive and offers

148 Guachun-Arias and Serpa-Andrade

Table 1. Syntax comprehension response results.

Table available in the original PDF paper.

Invocation name Type Results Star Motor Short Ok Stop Motor Short Ok Frequency Motor Short Ok

help through templates for the creation of customized skills (Agulló, 2021). Finally, it is necessary to install a database management system compatible with Phyton, Sqlite has been selected because of its autonomy and ease of use (Sandoval, 2021).

The final stage of the development is related to the interaction between the Raspberry board and the PLC. To program the PLC, the Logo Soft software is used, where control variables are configured, reading inputs and writing outputs, additionally for communication with Raspberry it is necessary to configure the network card to be visible.

On the other hand, it is necessary that the PLC has internet connection so that it can be connected to the Amazon cloud (AWS) from where the variables and input and output status can be monitored remotely, for this registration and configuration is performed in the cloud using the credentials of the PLC.

The PLC is programmed to send a control variable through an analog output, and in turn regulate the operating frequency of a drive, which in turn is controlling a motor.

TEST AND RESULTS This section is oriented to present the results achieved after the validation of the system against different parameters considered as priorities for the normal functioning of the system.

Validation of Request and Interactions The validation of the requests made is done by verifying the level of under- standing of Alexa for some specific invocation names, since in some cases it was not able to understand the command. In this sense, commands con- sisting of more than three words represent a higher degree of difficulty for Alexa to understand, while commands consisting of one or two words are easier for the virtual assistant to understand. In this way, a validation table of the environment is elaborated taking into account short and long com- mands to determine the level of understanding of Alexa. In some cases, it was verified that when long invocation names are used, the request is lost and Alexa returns to its idle state.

Response Times The validation of the response times is related to the level of understanding of Alexa to the invocation name, thus, it is evident that high response times are the result of a late understanding of the assistant to the order. Generally,

Implementation of an IoT-Based Environment 149

Table 2. Results of response times.

Table available in the original PDF paper.

Invocation name Response Time (s) Type of application Star Motor 0.1 Send Stop Motor 0.1 Send Frequency Motor 0.3 Reception

Table 3. Range of noise pollution into industrial environments (Gamine, Da Silva, Robazzi, Sauzo and Falerio, 2010).

Table available in the original PDF paper.

Level Range Description 1 0 to 20 dB Human hearing Threshold 2 20 to 50 dB Communication develops easily 3 50 to 80 dB Communication develops with difficulty 4 80 to 110 dB Limit value for a working day 5 Greater than 110 dB Communication is impossible

response times are in the order of milliseconds, however, for other commands the response has taken longer.

Another factor that affects a timely and adequate response is related to the type of request, different latency levels have been evidenced for actuator actuation requests compared to orders requesting information of some vari- ables, the response of the latter takes a few milliseconds more in relation to the former.

The following is a comparative table where the response times have been measured according to the request made.

In addition to the above, during the tests it became evident that the response latency is directly related to the traffic present in the network to which the virtual assistant is connected. On the other hand, according to the manufacturer’s specifications, in order to guarantee immediate response times, it is necessary to keep the firmware versions of the assistants updated.

Validation of Contaminated Enviroments Although the system response has been analyzed taking as a reference the level of understanding and the response time, one of the factors that most affects the correct operation of the developed environment is the noise level present, typical of industrial environments (López, 2020). For this purpose, validations were carried out with different dB levels in order to determine the system response to this scenario. In the same way, a comparison was made between different manufacturers of virtual assistants, in the same scenario, to determine the most suitable device for this type of interfaces, for this purpose, additionally, the specifications of each manufacturer for these products were taken as a reference.

To make the comparison between different industrial noise environments, the following power and intensity scale in decibels (dB) was used as a reference.

150 Guachun-Arias and Serpa-Andrade

Table 4. Contaminated environments test results.

Table available in the original PDF paper.

Range Distance (m) Results 20 to 50 dB 0.5 Ok 20 to 50 dB 1 Ok 20 to 50 dB 2 Ok 50 to 80 dB 0.5 Ok 50 to 80 dB 1 Ok 50 to 80 dB 2 Ok 80 to 110dB 0.5 Ok 80 to 110dB 1 X 80 to 110dB 2 X

Specifically in industry, noise can manifest itself continuously as a result of the operation of engines, machines, conveyor belts, etc.; as well as in the form of impact or punctual noise caused by hammers or presses.

For our analysis, tests have been performed for levels between 2 and 4, and at different distances, using the Android application Sound Meter Pro, which with proper calibration and according to the work developed in (Coyago, Zambrano, Guerrero and Orellana, 2020) can be used as a measurement instrument that provides accurate results through standardized processes as an alternative to highly expensive equipment.

In Table 4, we present the results of the validation of the voice assistant response to different ranges of industrial noise, in addition to the validation at distances of 0.5, 1 and 2 meters. The results show that at high noise levels, above 80 dB, interaction is only possible if the operator is at a distance of less than 0.5 meters from the voice assistant. For greater distances, the operator will have to raise his voice pitch to levels above 80 dB, which implies that he will have to shout and strain his throat so that the assistant can hear him, which implies problems associated with the misuse of the voice that may lead to future injuries (Domínguez, López, Núñez, Portela, Vazquez, 2020).

On the other hand, it can be highlighted the fact that, for levels above 110 dB where communication is impossible, it is not practical to implement this type of environment given the level of noise pollution.

Reduction of Operating Times The execution of the project was carried out in a laboratory environment; however, comparative tests were executed between the proposed environment and a typical control interface such as buttons. For the comparison, we used as a reference the set of frequency actuation value of a frequency inverter, where on the one hand the value is set by buttons and on the other hand the value is set by voice commands.

In addition, for any other type of information request in the traditional interfaces it is necessary for the operator to be in direct contact with the process, unlike with the virtual assistant he has the freedom of mobility within the coverage area.

Implementation of an IoT-Based Environment 151

CONCLUSION

The creation of “skills” for the virtual assistant is not limited to a specific syntax as long as they are related to the variable or actuator to be controlled, so that it is of common domain for different operators.

Although the use of virtual assistants is oriented for non-contaminated environments, these were tested in a noisy environment (industrial), where according to the validation tests, the system has limitations given the level of noise present in the environment, so that its use is not appropriate in the lower levels of the automation pyramid, however this limitation can be solved by using noise cancellation devices such as headphones or earphones.

On the other hand, the benefits of the system can be used at higher levels within the automation pyramid to request production history and alarms, and productivity projections.

The application of the project is focused on controlling the speed of a motor by adjusting the network frequency, however, the applicability of the IoT environment can be oriented to the control and monitoring of different sensors and actuators.

In general, the developed environment can be oriented to different areas of automation such as conveyor belts, tank filling control, among others. At the same time, it can be integrated to already implemented systems.

According to the reviewed bibliography, the authors have achieved the objective to control by voice commands both industrial processes and domo- tic systems, however, they lack a validation in polluted auditory environments typical of the industry.

REFERENCES

Álvarez Pedrón, J. (2020). Xana: Prototipo de asistente domótico controlado por voz (Doctoral dissertation). Agulló Valls, A. (2021). Gestión de Servicios de Amazon Web Services Mediante

Alexa. Balaguera Gómez, A. J. (2021). Estudio del proceso de automatización de las estaciones de trabajo de una línea de producción industrial (Bachelor’s thesis, Universitat Politècnica de Catalunya). Calero Gozálvez, C. (2021). Control de nivel de un sistema de tanques acoplados

mediante un autómata programable y una pantalla HMI. Carrera Hidalgo, Á. I. (2021). Desarrollo de una plataforma IoT para la supervisión y control de procesos industriales de fabricación inteligente en tiempo real a tra-vés de la nube (Master’s thesis). Chalán Padilla, V. A. (2020). Desarrollo de un controlador óptimo LQR utilizan-do

herramientas IOT para un sistema de presión constante controlado remotamen-te (Master’s thesis). Chuquimarca, L., Asencio, A., Torres, W., Bustos, S., Sánchez, J., & Saldaña, C.

(2022). Evaluation of Data Transfer from PLC to Cloud Platforms-Based Real- Time Monitoring Using the Industrial Internet of Things. In Information and Knowledge in Internet of Things (pp. 331-344). Springer, Cham. Coyago, A. S., Zambrano, M. M., Guerrero, M. L., & Orellana, D. (2020). Validación del uso de teléfonos inteligentes para medición de ruido ambiental urbano. Maskana, 11(2), 81–87.

152 Guachun-Arias and Serpa-Andrade

Diamantaras, I. (2019). Interfaz de usuario de voz para sistemas de Automa-tización. Domínguez-Alonso, J., López-Castedo, A., Núñez-Lois, S., Portela-Pino, I., & Vázquez-Varela, E. (2020). Perturbación de la voz en docentes. Revista española de salud pública, 93, e201908055. F Calvopiña, A., Tapia, F., & Tello-Oquendo, L. (2020). Uso del asistente virtual

Alexa como herramienta de interacción para el monitoreo de clima en hogares in- teligentes por medio de Raspberry Pi y DarkSky API. Revista Ibérica de Sistemas e Tecnologias de Informação, (36), 102–115. Ferreira Villabona, J. G., Hernandez Bernal, F. A., & Jaimes Suarez, y. M. (2020).

Diseño e Implementación de un módulo didácticode automatización industrial con interfaz hombre máquina hmi, variador de frecuencia y controlador lógico programable plc para el control de velocidad y par de arranque de un motor ac. Ganime, J. F., Da Silva, L. A., Robazzi, M. D., Sauzo, S. V., & Faleiro, S. (2010). El ruido como riesgo laboral: una revisión de la literatura. Enfermería global, 9(2). García, B. A. El paradigma actual de la subtitulación: cambios en la distribución de

contenido, nuevos hábitos de consumo y avances tecnológicos. La Linterna del Traductor, 53. López López, J. (2020). Estudio de rendimiento de asistentes virtuales de voz en condiciones ruidosas (Bachelor’s thesis). Manrique Hernández, J. A. (2018). Switch: un middleware para el desarrollo de

aplicaciones IOT con interfaces basadas en voz. Sandoval Pérez, J. G. (2021). Diseño e implementación de un video portero basa-do en raspberry pi (Bachelor’s thesis, Quito, 2021). Venegas, A. (2018). “IoT”: una nueva era, también para las marcas: Responsables de

agencias digitales hablan de como los anunciantes afrontan el internet de las cosas y de como la voz influirá en su desarrollo. Anuncios: Semanario de publici-dad y marketing, (1569), 22–24. Vicente-Samper, J. M., Campos, I., Sanz Worrel, B., Rodríguez, A., Oñate Tevar, J. M., & Sabater-Navarro, J. M. (2019). Ejemplo de integración de Alexa con un ro-bot UR. In XL Jornadas de Automática (pp. 360–365). Universidade da Coruña, Servizo de Publicacións. Zakharyan, E. (2018). Desarrollo de aplicaciones mediante robots colaborativos

basadas en interfaces naturales hombre-máquina (Doctoral dissertation, Universi- tat Politècnica de València).

Cover of the proceedings volume Production Management and Process Control

Explore the complete digital volume

Production Management and Process Control

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-12-4

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →