How do I find the ingest timestamp on Tier 1 and Tier 2?

Question

We work on a large System Platform solution (100k+ tags) where delays and interruptions are normal. Store-and-forward mechanisms are in place and we receive late data from time to time. 
 The system has two Tier 1 historians replicating to a common Tier 2 historian. 
 What is the best way to get the timestamp for when a data point was ingested/received/stored into Tier 1 and when it was replicated (received) on Tier 2?

Elliott Middleton · Accepted Answer

The system only maintains a timestamp from the source, not a timestamp of the ingestion. In cases where the source doesn't supply a timestamp, the system will apply a timestamp as close as possible to the source. For example, if a Communications Driver reads the timestamp from that PLC/RTU, that time will flow through across multiple levels of Replication; if that PLC/RTU sends the value without a timestamp, it will get a timestamp of "now" when it reaches the Communications Driver and that will be preserved. 
 If you're mostly wanting to measure the latency in Replication for the "happy case" (when everything is connected and streaming data), the only mechanism I can think of is to poll a replicated tag with a known source time stamp (e.g. SysTimeSec) at the "tier 2" system repeatedly with a query like this: 
 select DateTime, TagName, Value, QueryTime=getdate(), Latency=datediff(millisecond,DateTime,getdate()) from Live where TagName like 'MyTier1.SysTimeSec' 
 In my tests, "Latency" was consistently <3000 milliseconds for a tag that updates every second (so closer to 2000 milliseconds latency) in a system with no network latency and plenty of resources and very low data rates being replicated. You may have lower latency with higher data rates because buffers are sent when full or on a timer and the low data rates end up waiting for the timer.

Elliott Middleton · Answer

Handling the exception case isn't as straightforward (Niklas use case) because Replication "pushes" values by exception from the "tier 1", which makes it hard for the "tier 2" to detect when something wasn't pushed (does no value in 15 seconds mean the "tier 1" is offline or that the value just hasn't changed?). The only built-in instrumentation for this on the "tier 2" side are the system tags: 
 
 SysStatusTopicsRxData Total number of "Topics" (sources) receiving data 
 SysStatusSFDataPending Indicates that some HCAL clients have store-forward data to be sent to the server 
 
 The &ldquo;SysStatusSFDataPending&rdquo; tag (&ldquo;SF&rdquo; here is &ldquo;store forward&rdquo;) is a digital that may be of use. Here&rsquo;s a brief explanation of the state changes: 
 0 = None of the connected sources have store-forward data waiting to be sent or processed 1 = Some sources are connected which have store-forward data that is not yet available for retrieval 
 In this context, a &ldquo;source&rdquo; is Historian component that includes store-forward capability (e.g. a System Platform &ldquo;AppEngine&rdquo;, a "tier 1", a Plant SCADA Connector, a Remote IDAS, an application built with the Historian SDK). It is not the actual instrumentation that feeds that source (e.g. RTU, PLC, etc.) or the I/O server. 
 An integer tag, &ldquo;SysStatusTopicsRxData&rdquo;, indicates how many topics are currently connected. Because the Historian is mostly a passive service for remote sources, there is no standard way to know what this number &ldquo;should be&rdquo;, but in a given application you may know (e.g. &ldquo;there should always be 11 sources sending data&rdquo;). Within ~30 seconds of a topic's disorderly disconnect or faster for an orderly one, it will decrement. Here, "topics" includes the "sources" mentioned above and the I/O server topics. 
 These aren't as easy to use as we'd like for all the cases that may be important, though: 
 
 Volatile Sources : If there are a large number of sources sending data in a volatile network environment, &ldquo;SysStatusSFDataPending&rdquo; may be &ldquo;1&rdquo; much of the time because there is nearly always some store-forward data being processed while connections thrash in and out of store-forward. 
 Many Sources: If there are a large number of sources, it may not be useful to just know &ldquo;some source&rdquo; is offline or has pending store-forward data, because that would mean you can&rsquo;t really process any data. If the sources also had a &ldquo;heartbeat&rdquo; tag you can use to identify which individual one is offline, you might be able to work around this, but such a heartbeat would need to be part of the application (e.g. there isn&rsquo;t one built-in to Historian). In the case of Historian Replication, there are system tags on the &ldquo;tier 1&rdquo; which can help (SysReplicationSyncQueueItems1, SysReplicationSyncQueueValuesPerSec1), but you&rsquo;ll need to replicate them to be able to make use of them on the &ldquo;tier 2&rdquo; and also realize they will be &ldquo;stale&rdquo; if the source is disconnected or there is a store-forward still in progress. 
 When: These tags don&rsquo;t directly give any indication of what data was unavailable. You may be able to infer enough from when these tags change, though. For example, if &ldquo;SysStatusTopicsRxData&rdquo; decremented at &ldquo;10:17&rdquo; and incremented at &ldquo;11:09&rdquo; and that &ldquo;SysStatusSFDataPending&rdquo; became &ldquo;1&rdquo; at &ldquo;11:09&rdquo; and &ldquo;0&rdquo; at &ldquo;11:12&rdquo;, you may have enough information. As above, this gets more complicated when there are 20 such sources going in and out of store-forward at different times. 
 Latency: Where &ldquo;#3&rdquo; is less useful is when there is significant latency in the data. Using the example in &ldquo;#3&rdquo; above, if the data involved was stuck in an RTU for 4-hours before it made it into the Historian&rsquo;s source, the actual data that corresponds to the system tag changes would be for &ldquo;6:17&rdquo; to &ldquo;7:09&rdquo; and there isn&rsquo;t any information in the system which would indicate that.

How do I find the ingest timestamp on Tier 1 and Tier 2?

Top Replies