Plant SCADA 2023 R2 (U1), Deployment Server Process won't start, or will start slowly, or only deploy once.

Hi Dominic Xavier ,

Sorry to hear you are having problems with the Deployment feature.

It is great that you upgraded to Update 1, as this did have a number of quality improvements:

Deployment

Can you tell me a little about your project? Does your system have access to the Internet or DNS servers?

There have been previous issues where customer systems that had trouble contacting DNS, would cause delays.

I would recommend reaching out to AVEVA Customer Support and log a ticket with them to have this investigated further. I believe they will have a solution for you if it is related to DNS recursion. Quote issue number 3288728.

Also, before contacting Support, try collate the following information:

- Operations Control Management Console (Log Viewer) to see if any errors are reported on the Deployment Server?

- Windows Event Viewer.

- Logs folder, for the tracelog.SE.Asb.Deployment.Server.WindowsServer.dat file

Kind regards

Olivier

Hi Oliver,

I spotted that U1 resolved some issues but doesn't seem to have made any difference.

I can tell you a bit about the project:

The system has no access to the internet. It uses local DNS servers for name resolution and a local AD.

I am aware of the DNS recursion issue, and we have now tried with the DNS recursion on/off/NoNetworkTime variable set.

Unfortunately this made no difference.

In the process, however, we were collecting log files and noticed that the deployment server log file had information about repeated throwing exceptions "The remote name could not be resolved: 'computername'" (Computername used instead of actual name) yet all computers could be found by name on the network.

We copied the log file out and went to delete the ones in the folder and it wouldn't let us.

We stopped the deployment service and then deleted the log file.

Turned the service back on and haven't had a problem since. No further entries have been made (outside of normal startup) to the log.

The deployment server is operating normally.

Maybe something happened with the log file? Will keep an eye and follow up soon.

Cheers for your reply.

Thanks Eric, this seems like an odd and disappointing thing to need to do in 2024.  Our system should not have any such issues as it is all running on one VM host right now, but if it is having some communication issues, I would like to get to the bottom of those as it may well be impacting on other things.

Thanks Dominic.  Our customer is a bit old school with licensing, so hopefully that's not an issue for us.

I am not sure what you  mean about the SMS.  Do you mean it needs to be on the same machine or ....?

The main symptom I am seeing is that everything is slow to start, and presumably respond, eg this morning, the configurator started taking 3-4 mins to start!

Is your god user a domain user?

It's not a PlantSCADA problem. It's just a Windows workaround that should only be needed if your network is having trouble resolving computer names, which Dominic said he suspected was an issue with their old domain servers. I had to do that at one site which had deployment servers/clients on separate networks connected by a VPN but they had no domain controllers or DNS servers. On each local segment computers could resolve names, but not across the VPN to other segments.

Stuart,

We got the best performance with the system management server as a separate machine on the domain. I think Schneider recommend this but we did try having the SMS the same as the HMI server and the same as the license manager.

Yes, our god is a domain user. We also noted that the file permissions that Schneider set during the install should be domain tied, so join your computer to the domain FIRST, then install plantscada.

Our DNS issue had more to do with the DNS server net stack falling over. It didn't change the fact that the deployment server couldn't connect and was pushing items into the log. The log changed from names that couldn't be found, to IPs that couldn't be found. Can't recommend a lmhost file as I would not be the first person to forget about it and have weird things happen when IPs change, if you can't get to the DNS then hosts file will be your only option.

Configurator slow to start is a common issue. Still happens to us periodically. seems to be first restart. That might be worth another thread.

Thanks Eric, indeed, if we were in some sort of unreliable environment, then firming everything up with some hard coding would be a serious consideration, however I think in our case, as it is still all running in our lab, there is something more fundamental that is going on.  Perhaps it is config at some level, I am not sure.

Thanks Dominic, the SMS and deployment server are the only things on that server.  They did have a few more unnecessary components installed on it, but not configured, so I remedied that, to no avail unfortunately.  That process should have installed everything with the pertinent domain groups etc I would have thought, so I am still a bit stuck.

So I have now tried a hosts file on all of the VMs in the system, and it appears to be the same as before. I now have access to create a case with Aveva, so I Will do that.

 Hi Stuart Masters 
Nice to read from you.
We cannot use domain controller to host PCS and other component from Plant SCADA. By reading Dominic Xavier this coudl be his problem but we are mixing up.
Mi suggestion is to get in contact with your local distributor and raise the case.

I don't agree with the comment of Eric Black since the host file is only a solution to resolve DNS names. If the problem is accross domains we should have trusted domain (not necessary for all the plugs in that you want to use: AEL does, SMS and Deployment not). DNS suffix is a better and documented solutions for this.

There are the SMC logs that contain a lot of info when you need to troubleshoot.

It is something in your setup in my fair opinion so I will start to understand what it is timing up.

Regards

Hi  Stuart Masters 
The following fix will be available in Plant SCADA 2023 R2 U4 (October Update). Hope this update would resolve your issue.

3455908 - Deployment server performance impacted on a domain environment without an internet connection.
The deployment server configuration and its operation were slow when a client used a domain controller and both the client and domain controller were not connected to the internet. The Configurator will now launch quickly under these circumstances, and the deployment server configuration and deployment operations will respond promptly.

Thanks Jacky, I have gotten in touch with local Plant SCADA support (Schneider) and because of the other issues I was having on that machine, suggested KB41300 - and this somehow seems to have made a huge improvement.  I mean it fixed common services portal, which makes sense, but also somehow made configurator come up quicker and the deployment server service comes up quicker and the deployment activity has so far worked quicker and consistently. 

¯\_(ツ)_/¯

I am now monitoring the situation.

**Edit** deployment activity still times out, which makes more sense but unfortunately my problem persists, so hopefully it is the problem / fix you describe

Thanks Jacky, I have gotten in touch with local Plant SCADA support (Schneider) and because of the other issues I was having on that machine, suggested KB41300 - and this somehow seems to have made a huge improvement.  I mean it fixed common services portal, which makes sense, but also somehow made configurator come up quicker and the deployment server service comes up quicker and the deployment activity has so far worked quicker and consistently. 

¯\_(ツ)_/¯

I am now monitoring the situation.

**Edit** deployment activity still times out, which makes more sense but unfortunately my problem persists, so hopefully it is the problem / fix you describe