Database Not Connected Hardware Alarm - CitectSCADA 2018 R2 Update 4

I have been struggling with an issue here. In my hardware alarms, I have an alarm the states that the "Database Not Connected". Then when I go to the Alarm Summary page, I see thousands of alarms to say "Login attempt failed from <ip address> - unknown user" (the ip addresses on the message are my servers). When I look at my tracelog file, I see...

2020-03-22 10:47:27.747 -07:00 15492 0 Error AlarmClientAdaptor LegacyAdaptor::OnDataError ViewType=Display hCtrl=3 Error=DataRequestTimeout Message=Data not available Cluster=Stanton_U2
2020-03-22 10:47:27.747 -07:00 15492 0 Error AlarmClientAdaptor LegacyAdaptor::OnDataError ViewType=Display hCtrl=3 Error=DataRequestTimeout Message=Data not available Cluster=Stanton_U1
2020-03-22 10:47:27.747 -07:00 15492 0 Error AlarmClientAdaptor LegacyAdaptor::OnDataError ViewType=Display hCtrl=3 Error=DataRequestTimeout Message=Data not available Cluster=Stanton_U0

...when I look at my tracelog for the alarm server, I see...

2020-03-22 10:49:49.989 -07:00 15328 0 Error AlarmServerComms Exception An error occurred using the .NetApi Client in LogOn: {0} ClearScada.Client.AccessDeniedException: The username or password was incorrect.
at ClearScada.Client.Advanced.ScxComClient.ProcessServerException(Int32 requestCode)
at ClearScada.Client.Advanced.ScxComClientTcp.SendRequest(Int32 requestCode)
at ClearScada.Client.Advanced.ScxComLinkServer.LogOn(String userName, SecureString password, ILogonInformation& logonInformation)
at ClearScada.Client.Advanced.ScxComLinkServer.LogOn(String userName, SecureString password)
at ClearScada.Client.Simple.Connection.LogOn(String userName, String password)
at SchneiderElectric.Alarm.Server.Connection.Manager.ClearScadaClientApiConnection.LogOn(String userName, String password)

We have configured roles to use our corporate domain logins plus a few additional Citect users for the API connection used for the Wonderware Historian connector and kernal access.

We get these errors no matter what client we run, even the one on the servers. We also have shutdown the connector and all remote clients, same errors. I am beginning to think this is a bug of sorts, as these errors have added up to about 7GB of alarm event storage data in the last 12 days.

We have also tried to running the alarm servers in 64bit mode, same result.

We are running 2 physical servers, each with 3 clusters assigned to them. We have manually defined the port numbers for the second and third server processes so that they can coexist.

Being that we are run our clients and servers inside our own network, we have the windows firewalls turned off, but just for good measure, we have allowed all traffic on all ports and network types on both the servers and all clients.

What user name are the logs pointing to? We have setup the appropriate domain user groups to the Citect.**** groups. These errors still occur even if nobody in logged into the Citect client, it seems to be a server thing...but I'm not even sure that's accurate.

  • Wow, nice find.

    Could the difference be that this site has a domain and others are workgroups?
    Or perhaps there is a difference in Windows versions/updates or Citect patch levels?
    Just as much guessing as you are...
  • Hi Chirs,
    my bit here.
    I think you have your encryption missconfigured and using this user configuration (adding them in the Admin) made the workaround that you found.
    You can find the reason in the help:
    https://gcsresource.aveva.com/Citect/WebHelp/citect2018R2/Content/Operate_Runtime_Manager_in_Service_Mode.htm
    "
    Note: You will not be able to run your project if you are not a member of one of the following groups: Citect.Engineers, Citect.ServerUsers or Citect.LocalUsers. However, users with administrator privileges on the local machine can run a project even if they do not belong to one of these groups
    "

    So you need to check the encryption, the SMS, the win groups and the users.
    Maybe check with support.
  • Antonio,

    Thank you for the feedback, I have checked all that you have suggested before posting anything here. I even when as far as to re-install Citect on the servers. I have also tried turning encryption off completely with no change. It also didn't seem to matter if the servers. We have several server pairs thought our sites running this same version and no issues. That only real difference is this troubled site is running on server 2016 not 2012 like all the others.

    Thanks,

    Chris
  • An important note from the readme of 2018R2 (so other people can check in the future):

    As of the CitectSCADA 2018 R2 release date, Microsoft Windows Server 2016 does not support TLS 1.2 by default. You must enable it by applying Microsoft updates and several manual edits to the system registry. The tasks of applying all Microsoft updates and editing the system registry must be completed before you install CitectSCADA 2018 R2. These instructions also apply to any other software products that support TLS 1.2. Follow the instructions listed below.


    If you are required to enable TLS 1.2 and disable TLS 1.0 and TLS 1.1:


    1. Before installing CitectSCADA 2018 R2 on a Windows Server 2016 computer, make sure that your computer is up to date by downloading and installing all applicable Microsoft updates.
    2. If required by the updates, restart your computer.
    3. Edit the system registry. The .REG file shown below sets registry keys to their safest values. For additional information about these registry changes, see https://docs.microsoft.com/en-us/dotnet/framework/network-programming/tls#configuring-security-via-the-windows-registry.
    Windows Registry Editor Version 5.00
    [HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\.NETFramework\v4.0.30319]
    "SystemDefaultTlsVersions"=dword:00000001
    "SchUseStrongCrypto"=dword:00000001
    [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\v4.0.30319]
    "SystemDefaultTlsVersions"=dword:00000001
    "SchUseStrongCrypto"=dword:00000001
    4. Restart your computer to ensure that all changes take effect.
    5. Install CitectSCADA 2018 R2.
  • So, my issues have returned, I get the following errors in my trace-logs...this happen after I deployed an updated project.

    2020-04-02 08:28:40.626 -07:00 11892 0 Error Transport TcpipTransport::EndConnect() [CLIENT 0.0.0.0:61101 --> 10.29.0.41:22084 #48] SocketException: No connection could be made because the target machine actively refused it 10.29.0.41:22084

    2020-04-02 08:28:40.646 -07:00 11892 0 Error Transport TcpipTransport::EndConnect() [CLIENT 0.0.0.0:61063 --> 10.29.0.41:12080 #28] SocketException: No connection could be made because the target machine actively refused it 10.29.0.41:12080

    2020-04-02 08:28:40.699 -07:00 11852 0 Error Transport TcpipTransport::EndConnect() [CLIENT 0.0.0.0:61121 --> 10.29.0.41:12084 #52] SocketException: No connection could be made because the target machine actively refused it 10.29.0.41:12084


    It's like the servers have not opened their ports, but when I check for open ports using "netstat -q" in the command window, it shows that those ports are opened and can be connected to. Also, I ran Wireshark and found that both the servers and the clients are using TLS1.2, so I don't think it's an encryption issue, this happens with and without encryption enabled. We also have the Windows Firewall turned off on both the servers and clients. We can ping everything from anything, so not a switch or network issue.

    We have redundant NICs in the servers, and I have setup both IP addresses in the project for both servers and have assigned both network address names to each of my defined server process definitions.

  • Hi Chris,

    This is becoming a complex problem. I think the Support guys/girls are the best option for you right now.

    One thing you could check, that I could think of:
    Being able to ping from the client to the server does not guarantee that you can connect to the specific citect ports.
    You can use a telnet command from the client to try to connect to a specific server port.
    Naturally you won't be able to communicate anything useful with that, but it tells you if that port can be reached an thus it can be used as a sort of port specific ping command.
  • Patrick,

    I have done just that and will report that outcome of what we find so others can benefit on this thread.