Batch TrnExportCSV limitation for larger data

It's possible the trend data is corrupt in some way. You could try viewing it with Trend Reader. It should tell you if there's a problem. 

You could also try exporting that trend using Trend Reader or the TrnExportCSV Multi function. If the data files have some corruption, but the data (at least some of it) is still visible in the Citect runtime, TrnExportCSV Multi might be the best option since it exports data by reading it from the trend server in the runtime.

It's possible the trend data is corrupt in some way. You could try viewing it with Trend Reader. It should tell you if there's a problem. 

You could also try exporting that trend using Trend Reader or the TrnExportCSV Multi function. If the data files have some corruption, but the data (at least some of it) is still visible in the Citect runtime, TrnExportCSV Multi might be the best option since it exports data by reading it from the trend server in the runtime.

HI Eric, thanks a lot for the reply.

The Trend Reader is actually able to read the individual index files but not the HST file. When I try the HST file in Trend Reader, it provides me with the Error#9 Subscript out of Range window. The Batch TrnExportCSV tool also results in the Error #9 issue.

I have too much data to make going through each individual index file a viable option. Any additional options would be greatly appreciated.

Well, you could use TrnExportCSV Multi from the runtime, as I suggested, since the runtime apparently doesn't have a problem with that particular corruption. You could write a simple function to browse the trend tag list and call the export function for each tag.

Or, you could use my Trend HST Rebuilder utility to create new, valid, HST files, then export them with the Batch tool. The rebuilder can do a folder at a time, so that may save time.

Hi Eric can you provide the link to the toolbox and all these tools please

The links are in my previous messages

Hi Eric,

Using the tool to rebuild the HST file, I’ve now been able to use the bulk exporter to convert our data to CSVs. Thank you so much for your help.  

If we wanted to get away from having to do these conversions for future data we acquire while the machine is operating, what would be required? Is it as simple as installing a newer version of the Citect/AVEVA software on the local machine or something more complicated? We are currently in progress with a project to change how we store the data we obtain from the machine, if we could also improve how the data is stored (i.e. natively as a CSV or even on an SQL database) then it would really help us easily access the data going forward from now.

I'm glad to hear that worked.

I know there were some bugs in the trend system in very old Citect versions, like v5, where it could put invalid values in the HST under certain conditions (that was why I wrote the HST Rebuilder tool). If you have other old trend data files, then it's possible they would need the same fix. I haven't seen that kind of corruption in Citect v6 and later. Citect is very careful with how it writes to HST files so even if there's a power failure during a write, it normally doesn't hurt anything.

Citect only records trend data to proprietary trend data files. You could additionally record data to SQL or CSV files, but you'd have to write code (or a report) to do that. However, the trend pages in the project would only show data from the Citect trend data files.

If you want to store lots of trend data in SQL without having to write code, you could use Aveva Historian. Citect's trend pages would still only display data from the Citect trend files, but you could use the Aveva Historian Client on the Citect PCs to view historian data.

Another option is to use TrnExportCSVMulti Cicode so you could export trend data from the runtime to CSV files on demand. It just depends on whether you want to export some tags occasionally, or if you want all trend data to always be available in a SQL database that you can query.

If you do decide to use Aveva Historian, you could use the Import Citect Trends Into WW Historian tool to backfill all the existing trend data, since Historian only records new data.

Eric Black Your last sentence about the Historian is not true.

There is a backfill mechanism that help at the startup of the connector to automatically request all the data that you have stored in the TREND files. This parameter needs to be changed at the first startup of the Connector.

Is that something new? I've talked to tech support about that before and they told me that it only backfills back to the date the connector was first started. Also, they said the backfill mechanism is not designed for more than a few days of data. I can verify that's true because a customer had their historian offline for a few months. When I got it running again, the Historian Connector tried to push through all the data it had buffered during that time and the Historian got bogged down and started reporting lots of errors. I had to delete the buffered data and use my Import Trends tool to backfill the months of data in a format the Historian could handle.

Within the Historian Connector for Citect, there is a parameter MaxBackfillRange, which defaults to 1 day, which on first startup of the Connector, it will attempt to backfill data from that point. Once running, any stop/start or interruptions to the Connector, it will request data from the last sample time recieved. It tracks this timestamp per Trend tag in the sqlite database. So in theory, you could shutdown the Connector for months. Then start it up, and it will try its best to backfill from that last known sample (months ago). It will take a very long time to fully backfill, as there are performance limits on it. It performs a TrnQuery one Trend tag at a time. So depending on the size of the system, it could take a long time to complete and catchup to current time.