• Replies 46 replies
  • Subscribers 583 subscribers
  • Views 1862 views
  • Users 0 members are here
Options
Related

*** Recall of Citect SCADA 2018 / 2018 R2 Updates (July - October 2020)

Brad

Hi everyone, 

It's with regret that I share that we have identified a regression in recent Citect SCADA Monthly Updates that can affect the operational view of Alarm information including Alarm Summary, Alarm Counts, and other Alarm-related views, and it’s important that we notify our customers who could potentially be affected.

We have shared this message with potentially affected customers who have downloaded any of the following Monthly Updates;

  • Citect SCADA 2018 Update 22 (Jul 2020)
  • Citect SCADA 2018 Update 23 (Aug 2020)
  • Citect SCADA 2018 Update 24 (Sept 2020)
  • Citect SCADA 2018 Update 25 (Oct 2020)
  • Citect SCADA 2018 R2 Update 08 (Jul 2020)
  • Citect SCADA 2018 R2 Update 09 (Aug 2020)
  • Citect SCADA 2018 R2 Update 10 (Sept 2020)
  • Citect SCADA 2018 R2 Update 11 (Oct 2020)

 

The set of steps that produces this issue is complex, and involves systems including Alarm Redundancy, a large number of Clients, and events which are causing the Alarm Servers to switch roles and Display Clients to switch their connection. This can occur with cases of Domain Users and/or local Citect users. Most concerning, the occurrence of this can appear to be random, with the outcome being that the Alarm Counts, Alarm Summary and/or Alarm view components may fail silently. 

With our strongest recommendation we advise the following actions;

  1. If you haven’t yet implemented any of these Updates, postpone the installation.
  2. If you have implemented any of these Updates, uninstall these Updates and roll-back your implementation to the June 2020 Update, available from the AVEVA Knowledge & Support Center website at https://softwaresupport.aveva.com/. The Uninstallation process needs to be performed individually for each Update listed above.
  3. These Updates should be considered “recalled”; please delete any local copies so they’re not accidentally installed in future.
  4. If you have shared any of the above Updates with someone else, please also share this message with them.

 

For a Redundant configuration, our recommended order would be to first roll-back Primary Servers (across all Clusters, if required), followed by Clients, and finally any Standby Servers.

If you have any questions or concerns regarding the roll-back procedure for your Citect SCADA system, please reach out to your local Distributor and/or AVEVA Support team for specific advice – they will be your best contact for any questions.

Our investigations into the root cause of this issue are ongoing, and we will be publishing a new Update to resolve this issue as soon as possible. Unfortunately, I’m not yet able to forecast when a new Update might be available.

We have also taken the decision to delay the release of the November Update for both Citect SCADA 2018 & 2018 R2 versions as we prioritise the investigation and resolution of this issue, which is our highest development priority. I’ll provide further updates when I have more news regarding this investigation and the availability of a new Update.

I sincerely apologise for the inconvenience.

Kind Regards,

Brad Shaw
Product Manager – SCADA/HMI

  • Having spoken to our customer, due to the risk of rolling back 22 installs the decision was made to sit it out and wait for the update.
    We did however update our ini file in line with Geir's priority suggestions and I do believe it has made a bit of an improvement in repeatability of connecting to multiple clustered alarm servers (still problematic though). Thanks Geir!
  • In some cases where the rollback of the update is not possible or not desirable, a temporary alternative is to create a workaround mechanism to restart the Standby Alarm Server every hour automatically. You can put in place this temporary workaround till the newer patch is released.
    The applicable workaround is:

    1) For a multi-process system, add an event that will execute the ServerRestart("<AlarmServerName>","<ClusterName>"). Enable this event in the Standby Alarm Server process exclusively.
    For more info see the help: Cicode Programming Reference > Cicode Function Categories > Server Functions > ServerRestart.
    2) For a single-process system, add an eventProcessRestart(). Enable this event in the Standby Server machine exclusively.
    For more info see the help: Cicode Programming Reference > Cicode Function Categories > Miscellaneous Functions > ProcessRestart.
  • If you still have issues, you may use the solution that I proposed in my previous comment.
  • This has been published in this Tech Alert:
    https://softwaresupportsp.aveva.com/#/okmimarticle/docid/ta482
  • Would this not mean that, if the primary server failed, the only alarm server would be restarting each hour?
    Is as frequently as hourly necessary?
  • No, this means that the issue you can workaround it only by restarting the Standby Alarm server.
    With a couple of sites, after applying the workaround, we didn't see any problem.
    I think one hour is a good compromise.
  • Aveva Support

    is there any update on an ETA for this issue correction ?

  • For what it's worth, we have had this issue occur on both servers (many many times); restarting the affected server fixes the problem with the problem clients, forcing a failover to the "other" server.

    What we found is the problem will only show on the least loaded server; as in the one with the least number of active connections (your mileage with this advice may vary!).

    Given clients with default settings "stick" to the server they last swung to - unless that the server is rebooted or unless starting from cold (client starts with the primary) - it's quite possible for the primary sever to show the problem. By restarting just one of the servers you will force the clients onto the other server - where at least on our site - the most active server runs fine. We had it mostly on our primary server because we rebooted the primary last when deploying. The primary had no clients until the clients where rebooted and swung back. The bug sometimes occurred straight away after a deploy or after 5 days.

    Another way we caused the bug on a connected client / upset the alarm server is to perform a query through the Alarm server ODBC backend. We have had many cases occur due to running these queries; note: random occurrences, not every time. We greatly improved by turning the queries off (custom alarm historian).

    As I understand it due to nature of the Update8-11 bug, be aware that the clients may start directly on the standby. So best to check "Page Table Tran" on your Alarm server process to confirm what is actually going on. 

    For those having trouble uninstalling, you simply need to point that uninstall browse window at a valid 2018R2 source directory with the MSI. ie.<Some path to your install files>\Citect_SCADA_2018_R2\Citect SCADA 2018 R2\Citect\Citect Scada 2018 R2.msi

    I've had machines that have had multiple patches installed - old patches appear from "escrow" after you remove a higher one - if the uninstall browse for install files popup shows doesn't seem to follow rhyme nor reason ! For us it might fail on removing Update 10 or Update 1 or not at all. I can confirm though, that every time I have supplied a valid path to the MSI, the uninstall works correctly !

    Finally,  thanks for your candor and being above board like this. Update 7 gave us immediate relief which is a massive benefit. 

  • Any ETA on the November Update as yet to correct this issue ?
  • Hi ,
    We have identified the code change that caused the regression in the alarm system. We are now putting the November Update through a test cycle of regression and system testing. We are hopeful that early next week we should have the testing complete the the Update available. We will keep you posted if there is any change to this plan.
© 2024 AVEVA Group Limited or its subsidiaries. All Rights Reserved.