Plant SCADA 2023 R2 (U1), Deployment Server Process won't start, or will start slowly, or only deploy once.

Hi Dominic Xavier ,

Sorry to hear you are having problems with the Deployment feature.

It is great that you upgraded to Update 1, as this did have a number of quality improvements:

Deployment

Can you tell me a little about your project? Does your system have access to the Internet or DNS servers?

There have been previous issues where customer systems that had trouble contacting DNS, would cause delays.

I would recommend reaching out to AVEVA Customer Support and log a ticket with them to have this investigated further. I believe they will have a solution for you if it is related to DNS recursion. Quote issue number 3288728.

Also, before contacting Support, try collate the following information:

- Operations Control Management Console (Log Viewer) to see if any errors are reported on the Deployment Server?

- Windows Event Viewer.

- Logs folder, for the tracelog.SE.Asb.Deployment.Server.WindowsServer.dat file

Kind regards

Olivier

Hi Oliver,

I spotted that U1 resolved some issues but doesn't seem to have made any difference.

I can tell you a bit about the project:

The system has no access to the internet. It uses local DNS servers for name resolution and a local AD.

I am aware of the DNS recursion issue, and we have now tried with the DNS recursion on/off/NoNetworkTime variable set.

Unfortunately this made no difference.

In the process, however, we were collecting log files and noticed that the deployment server log file had information about repeated throwing exceptions "The remote name could not be resolved: 'computername'" (Computername used instead of actual name) yet all computers could be found by name on the network.

We copied the log file out and went to delete the ones in the folder and it wouldn't let us.

We stopped the deployment service and then deleted the log file.

Turned the service back on and haven't had a problem since. No further entries have been made (outside of normal startup) to the log.

The deployment server is operating normally.

Maybe something happened with the log file? Will keep an eye and follow up soon.

Cheers for your reply.

Hi Dominic Xavier ,

Thanks for the update. Definitely sounds like a gremlin in the system was causing an issue. I am not sure why the logging would cause an issue with the Deployment Server. It is more likely the act of restarting the service got it out of the bad state.

Keep an eye on it and hopefully it is more reliable going forward.

Kind regards

Olivier

Hi Oliver,

I completely agree that the service restart is the fix there, but previously it would never allow us to start the service... and if it did it would only work for a short time. Once the log file was deleted and the service restarted it behaved normally.

Keeping an eye on it, and keeping an eye on what it puts in the log file. If there is any further updates I'll post.

Thanks for your response.

Hi Dominic Xavier , we are having a similar sounding problem, but are already at update 2 and I have deleted the log files (there were some weird errors in there) - now I just have an error that my engineering machine where the deployment client is running is offline (which it's not) and when I go onto that machine and try to view computers or versions the requested operation pretty much always times out.  Just wondered if any of this resonates with you and if I am missing some critical step(s)

I look forward to hearing from you, 

Stuart, deployment server still gives us grief at times.

We have found: SMS choice makes a huge difference. We found it works best when the SMS is NOT a server nor a client to the deployment system.

For whatever reason, name resolution is still a problem from time to time. We are using older domain servers (2012 I think?) and we suspect this may be part of the problem, however can't verify. Log files fill up with "cant connect to COMPUTERNAME" when the DNS is having problems, and they fill up with "can't connect to 192.168.73.23" when it's operating normally. Restart your DNS server, then restart your deployment server and delete the logs again, check if it showing up with name or IP. if its still name in the log files, try again?

Security - create a god user and add them to all PlantSCADA roles, log everyone else out and log in as god on deployment server and client.

Firewall - Plant SCADA says it modifies the firewall, but for some reason on ours it did not add the domain network to the firewall rules - check.

If you are using the new fancy pants licensing - put your license server and license manager on a separate computer to your deployment server. Port 443 is in hot contention.

I'd be interested to hear how you go.

Stuart, deployment server still gives us grief at times.

We have found: SMS choice makes a huge difference. We found it works best when the SMS is NOT a server nor a client to the deployment system.

For whatever reason, name resolution is still a problem from time to time. We are using older domain servers (2012 I think?) and we suspect this may be part of the problem, however can't verify. Log files fill up with "cant connect to COMPUTERNAME" when the DNS is having problems, and they fill up with "can't connect to 192.168.73.23" when it's operating normally. Restart your DNS server, then restart your deployment server and delete the logs again, check if it showing up with name or IP. if its still name in the log files, try again?

Security - create a god user and add them to all PlantSCADA roles, log everyone else out and log in as god on deployment server and client.

Firewall - Plant SCADA says it modifies the firewall, but for some reason on ours it did not add the domain network to the firewall rules - check.

If you are using the new fancy pants licensing - put your license server and license manager on a separate computer to your deployment server. Port 443 is in hot contention.

I'd be interested to hear how you go.

If your domain controllers are not reliably reachable, you may need to create a hosts file on the servers and clients so they can reach each other for deploying projects.

Edit C:\Windows\system32\drivers\etc\hosts file in Notepad (you may need to run Notepad as administrator). Add the names and IP addresses of each server and client. Save the file with no extension. Copy the file to each server/client so they can all find each other.

Thanks Eric, this seems like an odd and disappointing thing to need to do in 2024.  Our system should not have any such issues as it is all running on one VM host right now, but if it is having some communication issues, I would like to get to the bottom of those as it may well be impacting on other things.

Thanks Dominic.  Our customer is a bit old school with licensing, so hopefully that's not an issue for us.

I am not sure what you  mean about the SMS.  Do you mean it needs to be on the same machine or ....?

The main symptom I am seeing is that everything is slow to start, and presumably respond, eg this morning, the configurator started taking 3-4 mins to start!

Is your god user a domain user?

It's not a PlantSCADA problem. It's just a Windows workaround that should only be needed if your network is having trouble resolving computer names, which Dominic said he suspected was an issue with their old domain servers. I had to do that at one site which had deployment servers/clients on separate networks connected by a VPN but they had no domain controllers or DNS servers. On each local segment computers could resolve names, but not across the VPN to other segments.

Stuart,

We got the best performance with the system management server as a separate machine on the domain. I think Schneider recommend this but we did try having the SMS the same as the HMI server and the same as the license manager.

Yes, our god is a domain user. We also noted that the file permissions that Schneider set during the install should be domain tied, so join your computer to the domain FIRST, then install plantscada.

Our DNS issue had more to do with the DNS server net stack falling over. It didn't change the fact that the deployment server couldn't connect and was pushing items into the log. The log changed from names that couldn't be found, to IPs that couldn't be found. Can't recommend a lmhost file as I would not be the first person to forget about it and have weird things happen when IPs change, if you can't get to the DNS then hosts file will be your only option.

Configurator slow to start is a common issue. Still happens to us periodically. seems to be first restart. That might be worth another thread.

Thanks Eric, indeed, if we were in some sort of unreliable environment, then firming everything up with some hard coding would be a serious consideration, however I think in our case, as it is still all running in our lab, there is something more fundamental that is going on.  Perhaps it is config at some level, I am not sure.

Thanks Dominic, the SMS and deployment server are the only things on that server.  They did have a few more unnecessary components installed on it, but not configured, so I remedied that, to no avail unfortunately.  That process should have installed everything with the pertinent domain groups etc I would have thought, so I am still a bit stuck.

So I have now tried a hosts file on all of the VMs in the system, and it appears to be the same as before. I now have access to create a case with Aveva, so I Will do that.

 Hi Stuart Masters 
Nice to read from you.
We cannot use domain controller to host PCS and other component from Plant SCADA. By reading Dominic Xavier this coudl be his problem but we are mixing up.
Mi suggestion is to get in contact with your local distributor and raise the case.

I don't agree with the comment of Eric Black since the host file is only a solution to resolve DNS names. If the problem is accross domains we should have trusted domain (not necessary for all the plugs in that you want to use: AEL does, SMS and Deployment not). DNS suffix is a better and documented solutions for this.

There are the SMC logs that contain a lot of info when you need to troubleshoot.

It is something in your setup in my fair opinion so I will start to understand what it is timing up.

Regards

Hi  Stuart Masters 
The following fix will be available in Plant SCADA 2023 R2 U4 (October Update). Hope this update would resolve your issue.

3455908 - Deployment server performance impacted on a domain environment without an internet connection.
The deployment server configuration and its operation were slow when a client used a domain controller and both the client and domain controller were not connected to the internet. The Configurator will now launch quickly under these circumstances, and the deployment server configuration and deployment operations will respond promptly.