SOE Filtering Speed for Specific Alarms (Citect 2018 R2)

Hi Community,

We were planning on using the SOEEventAdd() function to provide the plant operators with a method of attaching a 'note' (ie. a simple string) to a selected piece of equipment, and have this viewable/editable on a faceplate when that piece of equipment is selected.  Using the SOE (alarm summary) would also provide for a history of attached notes and the inbuilt archiving functionality of the standard alarm database.

Unfortunately, whilst the above functionality all works, the time it takes to filter for a specific alarm tag is a cause for concern, and may lead to the functionality being scrapped.  Our current calculations show that filtering on a specific alarm tag (see below) will only process summary events at the rate of about 25,000 records per second.  If our (small) plant generates 5 summary events per minute, then it will have generated over 100,000 events in 2 weeks.  Using such a filter will take 4sec to return any data.  Longer periods appear to scale linearly (eg. 4 weeks of events takes 8 sec to filter).

iHndl = AlarmFilterEditOpen(hAn);

IF iHndl <> BAD_HANDLE THEN

AlarmFilterEditSet(iHndl, "TAG=" + sEquipment + "_Note");

AlarmFilterEditCommit(iHndl);

AlarmFilterEditClose(iHndl);

END

The above code is very much derived from the SA_Include\InfoAlarm.ci function called InfoAlarm_EquipmentFilterTask, with the exception that this is a Summary Alarm list (type 15).

My question (after this exceptionally long preamble) is, are there any optimized methods of extracting lists of specific SOE events from the alarm summary?

 

Some other things I've tried (and have just resulted in very slow alarm list display; category and priority 73 were specific to the operator notes):

   // AlarmFilterEditSet(iHndl, "CATEGORY=73");
   // AlarmFilterEditAppend(iHndl, ";EQUIPMENT=" + sEquipment);

   // AlarmFilterEditSet(iHndl, "PRIORITY=73");
   // AlarmFilterEditAppend(iHndl, ";EQUIPMENT=" + sEquipment);

   // AlarmFilterEditSet(iHndl, "EQUIPMENT=" + sEquipment);
   // AlarmFilterEditAppend(iHndl, ";TAG=" + sEquipment + "_NoteLog");

The only thing that really works is a hard date/time limit, but that sort of defeats the purpose of the whole thing.

   AlarmFilterEditSet(iHndl, "TAG=" + sEquipment + "_NoteLog");
   AlarmFilterEditAppend(iHndl, ";DATE>=" + IntToStr(TimeCurrent()-14*86400));

 

  • Hi Steve,

    This is partially the nature of SOE; events are stored in 1 hour files and objects (alarms) are grouped to gather per 256 (default streamsize 8).
    The more alarms you have in your system the more spread the events will become and the larger the amount of data to be possibly queried resulting in longer periods.

    For your case :
    - Consider specifying the Alarm_Note on a separate alarm-only cluster (call it "EquipNotes" for now)
    This allows you to tweak the performance a lot better without interfering with your normal alarms SOE and Alarm Summary
    - You will have to specify an additional Alarm server pair for this (think about the ports to use and possible firewalls if on same server as other alarm process)
    - On the display for the history you can then use 'AlarmDsp(AN, Count, 15, "EquipNotes"))'
    so it will only connect to this Notes Cluster
    - Then you can filter for the specific notes using the AlarmFilterEditSet for the tag

    One downside is that on all your existing alarmpages / list you need to specify the existing cluster as well (in the function or in Page Cluster Context)
    Next is, this approach doesn't really combine well with your existing Equipment; to add the equipment "tree" you would have to define it duplicate on a separate cluster; messy.

    Good thing is you could create multiple categories using this for e.g Equipment notes, but also have dummy alarms to record e.g Shift Notes and easily reuse existing alarm templates for displaying all notes from this specific cluster.

    If you still go for this approach, to tune this EquipNotes alarm process optimal:
    - Only make modifications to '[Alarm.<ClusterName>.<ServerName>]' section so [Alarm.EquipNotes.<ServerName>] so you are not interfering with your normal alarms
    - [Alarm.EquipNotes.<ServerName>]DisableSummary=1 -> you are only using the SOE for this one
    - [Alarm.EquipNotes.<ServerName>]StreamSize=n ->set it big enough to have all events in one file per hour. Number of objects = 2 to the power n
    - [Alarm.EquipNotes.<ServerName>]CacheSize=250 -> this is about caching queries in memory for re-use if memory in the server is sufficient, set this higher. Also depends on alarm server setting for "Extended Memory" (64bit)
    - [Alarm.EquipNotes.<ServerName>]KeepOnlineFor=x -> max number of weeks you want to keep the notes ( the longer you make this the more the query might slow down over time)
    - [Alarm.EquipNotes.<ServerName>]ArchiveAfter=1 -> Assuming you are not filing Timestamp older than 1 week in SOEEventAdd calls

    I did a mock up / demo for this once, but have not tried it in production, so any feedback is welcome.
  • Thanks for the response Erik.

    Interesting idea about running two alarm clusters. We'll have to look into the ramifications of splitting the system, as it was (mostly) designed to handle multiple clusters but has never been tested in such a configuration.

    Your comment about the journaling structure (streamsize) was interesting though. I had imagined that if I could force the operator note alarms into different journal files than the standard alarm summary events that this might affect the query response time. Unfortunately, this does not appear to be the case. My test system has 113 directories for journal history files (JnI00000 to JnI00112), and I've generated (overnight) 300,000 standard alarm summary events that fall into JnI00082. The operator note alarms fall into JnI00104-JnI00112, and yet a query for a specific tag lying in one of those journals takes over 17sec to return.

    It would be nice to know quite a bit more information about how the alarm sever works and whether there are any optimisations inherent in the system that we could lever off to get the performance that we require.
  • Hi Steve,

    You cannot really force the operator notes into a separate stream. This is based on the underlying unique object Id generated at object creation. Only sort of way to achieve this is to add them "later" (so compile - start - run once without the alarm notes and then add them, so they get appended after blocking them all together). Based on your remark JnI00104-JnI00112 this is hat sort of happened anyway.

    When you say query takes 17 sec, I reckon this is measuring using the AlarmDsp or AlarmListCreate functions.
  • Erik,

    Yep; once I figured how the journal streams were being created, I deliberately modified the note alarms to push them to the end of the journal blocks. If the test had worked then I might have been tempted to go back to support and ask them to journal specific* SOE alarms into separate blocks (eg. JnZ00001->), but the server's doing something else, so no point.
    * probably based on the SOE alarm classification

    Steve