Alarm Server in Extended Memory Running as a Service Citect 2016

Hello

I have a redundant system (2 servers). Citect 2016. For this situation one server is running as a service and the other is not (to have better debugging) 

The alarm servers are taking care of about 30k digital alarms and some advanced.

This has been working for years without problems but recently I add some motors and more 2k digital alarms and one of the alarm Servers crashed (at night of course) with out of memory message in the log. I check the memory used by the alarm servers and they start about 900Mb and grow until about 1.2 Gb (primary and standby is the same)

One of the things I tried to solve the situation was to activate the "Extended Memory" in both alarm servers, but the one running as a service doesn't start with configuration error or something similar.

Someone is using Extended Memory in a alarm server running as a service? 

Any other advice for the out of memory problem (other than delete alarms, add a new alarm server, implement clusters)

Parents
  • Have you tried to start the alarm server as service first? Have a look at the trace/sys log of this alarm server and see any suspected errors reported. It is worth trying on the alarm server as service when the other alarm server starts first

    • Stop Runtime Manager service
    • Back up the alarm data files from your data folder
    • Delete alarm files folder
    • Restart Runtime Manager service

    The alarm data on this server should be synched from the other server.

    Also ensure both alarm servers have the same version/update and configured with the same set of alarm parameters, such as [Alarm]ArchiveAfter and KeepOnlineFor.

    regards,

    jacky

  • Thanks  
    Both alarm servers have the same parameters, the system is redundant and is on production so I can't start this server before the other, as to do that they need to be both stopped, maybe for Christmas.

    The error is mismatch with extended memory property.

    Do you have your AlarmServer running with extended memory as a service in Citect 2016?

  • The screenshot of your alarm syslog reveals that the alarm process in the extended memory mode attempted to run in 32 bit process architecture. Check the Windows version of this target machine. If the Windows is 64 bit, there could be something wrong with Runtime Manager that fails to launch Citect.exe (64 bit) process. If it is this case, please reach out to Technical Support for troubleshooting.

    [32 bit process start up]
    2022-04-11 08:40:15.552 +10:00 *** SCADA process is starting up ***
    2022-04-11 08:40:15.552 +10:00 Running in 32 bit process architecture

    [62 bit process start up]
    2022-11-01 09:56:04.163 +11:00 *************************************
    2022-11-01 09:56:04.164 +11:00 *** SCADA process is starting up ***
    2022-11-01 09:56:04.164 +11:00 Running in Extended Memory mode (64 bit) process architecture

  • Thanks Jacky

    It seams that the restart of the alarmServer process didn't change it from 32bit to 64bit, but stopping all processes and starting all processes again will work well.

    Case solved.

  • It appears you must shutdown/startup Runtime Manager for these changes to take effect. It is not possible to restart individual processes. Glad you found a solution to your problem!

  •  thanks for the feedback, we may want to add this in a TN.

  • This is just an additional note. It's solved. It's obvious, but I had some time around it!

    Some of the alarm categories have the storage / summary device as SQL device.(DSN=scada)

    The database is MySQL with MariaDB ODBC driver. (the DSN is configured as "System DSN" so the service can reach it)

    This configuration was running well when the AlarmServer was not running in protected mode, but now I need it running in protect mode, that works in 64bit.

     After change to protected mode I get in debug.Alarm.CLUSTER.AlarmServer.log several errors:

    “SQL Error execution error 307 for SQLDeviceOpen with handle 999 in UsrSQLUniversalPost(…)” when I stat the server

    “SQL Error execution error 274 for SQLDeviceAppend with handle 999” when there is an alarm to record in the summary

    And nothing is written in the database.

    IThe error 274 probably is because the handle to the SQL is not open, The error 307 refers to SQLErrMsg, but I can't call it as the SQLDeviceOpen is called internally by the AlarmServer.

    installed the 64bit ODBC driver and create a 64bit System DSN with same name using the 64bit driver (in both servers), but the problem continues.

    Changed the DSN of the 64bit to a different name from the 32bit DSN and it started working fine.

Reply
  • This is just an additional note. It's solved. It's obvious, but I had some time around it!

    Some of the alarm categories have the storage / summary device as SQL device.(DSN=scada)

    The database is MySQL with MariaDB ODBC driver. (the DSN is configured as "System DSN" so the service can reach it)

    This configuration was running well when the AlarmServer was not running in protected mode, but now I need it running in protect mode, that works in 64bit.

     After change to protected mode I get in debug.Alarm.CLUSTER.AlarmServer.log several errors:

    “SQL Error execution error 307 for SQLDeviceOpen with handle 999 in UsrSQLUniversalPost(…)” when I stat the server

    “SQL Error execution error 274 for SQLDeviceAppend with handle 999” when there is an alarm to record in the summary

    And nothing is written in the database.

    IThe error 274 probably is because the handle to the SQL is not open, The error 307 refers to SQLErrMsg, but I can't call it as the SQLDeviceOpen is called internally by the AlarmServer.

    installed the 64bit ODBC driver and create a 64bit System DSN with same name using the 64bit driver (in both servers), but the problem continues.

    Changed the DSN of the 64bit to a different name from the 32bit DSN and it started working fine.

Children
No Data