Hi all,
We are system integrators and one of our clients is using CitectSCADA 2016 with 5 operator stations (control clients) and 2 servers (redundant). The operators have noticed that after a certain time (can vary from a few hours to a few days), they are not able to send commands or change setpoint values anymore from their workstation but they are still receiving values and alarms so the station is still functional in "view" mode. When this happens, they have to reboot their workstation to "fix" the issue. They are still able to send commands from the other workstations so this doesn't seem to be a server issue.
To help in troubleshooting this, I asked them to notify me the next time this happened and they did so. I logged in remotely on their station and tried modifying a setpoint and it didn't change. I tried toggling a digital signal to ON and it didn't work. I then tried using TagDebug to modify the tag values and it still didn't work.
I tried the same changes directly on the display client process running on the server and they all worked. The tags I tried to write to from the operator workstation where either disk IO tags or OPC tags sent to a PLC.
I've looked at the log files from the workstation and I've noticed the following errors ("One or more tag operation callbacks are cancelled") which occurred at about the same times that the commands were not working. There is also a "3" logged before each of these errors and also errors about tasks taking more than 1 second to execute.
Could these errors be the cause of my problem? I'm suspecting either a timeout issue or delays when the workstation is sending the command to the server (maybe the IT department added new firewall rules or something that is slowing the communication down? They have been having this problem for a few months but the system was running fine before (since 2006) and since 2016 with this version of Citect. This problem is happening on either one of their 5 workstations so it isn't tied to a particular computer.
Any help would be greatly appreciated.
Thank you,
Patrice Jacob
SCADAlogik inc.
