Plant SCADA 2023 R2 (U1), Deployment Server Process won't start, or will start slowly, or only deploy once.

Been having problems with a new (updated) install of PlantSCADA. 20 clients, redundant server, on a domain.

Configured deployment server on development machine. Deployment logs in, creates a deployment and allows deployment to clients and servers, and mostly crashes mid way through deployment (although appears to complete) and then can never deploy again.

Updated to Update 1 with same results. Have tried multiple options (with/without encryption, win11 development machine and server 2016 development machine, move SMS machine, etc no changes.)

Client authenticates to server correctly. SMS, and licensing are on a separate machine. Standard ports.

You can create deployments and deploy once (when lucky) otherwise the process either stops or stops responding. Services says its running, but studio gets stuck in 'refreshing' for computers and versions. Restart the computer and about 50% of the time it starts straight away. The other half the service never starts, just get stuck at 'starting'.

Any ideas welcome.

Parents
  • Hi  ,

    Sorry to hear you are having problems with the Deployment feature.

    It is great that you upgraded to Update 1, as this did have a number of quality improvements:

    Deployment

    2864853 - Deployment activity keeps refreshing when using a domain user

    The deployment authorisation logic has been simplified to improve performance and correct a token timeout calculation that would cause constant refreshing.

    3047818 - Deployment status disappears from the deployment server Computers view

    During deployment of large projects, if the deployment operation went beyond a user session timeout interval it would not be successful due to the user session becoming unauthorized. Now when a deployment operation commences, it will remain active even after the user session expires.

    2968800 - Target and Profile columns on the Computers view occasionally revert to their previous values

    A background operation to refresh the Computers view of the Deployment screen will now be suppressed after the Target or Profile dropdown control has been pressed. The background operation will be restored when the process completes.

    3290142 - Deployment Server password unable to contain double quotes and semi-colon characters

    The Deployment Server has been updated to permit passwords containing double quote and semi-colon characters.

    Can you tell me a little about your project? Does your system have access to the Internet or DNS servers?

    There have been previous issues where customer systems that had trouble contacting DNS, would cause delays.

    I would recommend reaching out to AVEVA Customer Support and log a ticket with them to have this investigated further. I believe they will have a solution for you if it is related to DNS recursion. Quote issue number 3288728.

    Also, before contacting Support, try collate the following information:

    - Operations Control Management Console (Log Viewer) to see if any errors are reported on the Deployment Server?

    - Windows Event Viewer.

    - Logs folder, for the tracelog.SE.Asb.Deployment.Server.WindowsServer.dat file

    Kind regards

    Olivier

  • Hi Oliver,

    I spotted that U1 resolved some issues but doesn't seem to have made any difference.

    I can tell you a bit about the project:

    The system has no access to the internet. It uses local DNS servers for name resolution and a local AD.

    I am aware of the DNS recursion issue, and we have now tried with the DNS recursion on/off/NoNetworkTime variable set.

    Unfortunately this made no difference.

    In the process, however, we were collecting log files and noticed that the deployment server log file had information about repeated throwing exceptions "The remote name could not be resolved: 'computername'" (Computername used instead of actual name) yet all computers could be found by name on the network.

    We copied the log file out and went to delete the ones in the folder and it wouldn't let us.

    We stopped the deployment service and then deleted the log file.

    Turned the service back on and haven't had a problem since. No further entries have been made (outside of normal startup) to the log.

    The deployment server is operating normally.

    Maybe something happened with the log file? Will keep an eye and follow up soon.

    Cheers for your reply.

  • Hi  ,

    Thanks for the update. Definitely sounds like a gremlin in the system was causing an issue. I am not sure why the logging would cause an issue with the Deployment Server. It is more likely the act of restarting the service got it out of the bad state.

    Keep an eye on it and hopefully it is more reliable going forward.

    Kind regards

    Olivier

  • Hi Oliver,

    I completely agree that the service restart is the fix there, but previously it would never allow us to start the service... and if it did it would only work for a short time. Once the log file was deleted and the service restarted it behaved normally.

    Keeping an eye on it, and keeping an eye on what it puts in the log file. If there is any further updates I'll post.

    Thanks for your response.

  • Hi  , we are having a similar sounding problem, but are already at update 2 and I have deleted the log files (there were some weird errors in there) - now I just have an error that my engineering machine where the deployment client is running is offline (which it's not) and when I go onto that machine and try to view computers or versions the requested operation pretty much always times out.  Just wondered if any of this resonates with you and if I am missing some critical step(s)

    I look forward to hearing from you, 

  • Stuart, deployment server still gives us grief at times.

    We have found: SMS choice makes a huge difference. We found it works best when the SMS is NOT a server nor a client to the deployment system.

    For whatever reason, name resolution is still a problem from time to time. We are using older domain servers (2012 I think?) and we suspect this may be part of the problem, however can't verify. Log files fill up with "cant connect to COMPUTERNAME" when the DNS is having problems, and they fill up with "can't connect to 192.168.73.23" when it's operating normally. Restart your DNS server, then restart your deployment server and delete the logs again, check if it showing up with name or IP. if its still name in the log files, try again?

    Security - create a god user and add them to all PlantSCADA roles, log everyone else out and log in as god on deployment server and client.

    Firewall - Plant SCADA says it modifies the firewall, but for some reason on ours it did not add the domain network to the firewall rules - check.

    If you are using the new fancy pants licensing - put your license server and license manager on a separate computer to your deployment server. Port 443 is in hot contention.

    I'd be interested to hear how you go.

  • If your domain controllers are not reliably reachable, you may need to create a hosts file on the servers and clients so they can reach each other for deploying projects.

    Edit C:\Windows\system32\drivers\etc\hosts file in Notepad (you may need to run Notepad as administrator). Add the names and IP addresses of each server and client. Save the file with no extension. Copy the file to each server/client so they can all find each other.

Reply
  • If your domain controllers are not reliably reachable, you may need to create a hosts file on the servers and clients so they can reach each other for deploying projects.

    Edit C:\Windows\system32\drivers\etc\hosts file in Notepad (you may need to run Notepad as administrator). Add the names and IP addresses of each server and client. Save the file with no extension. Copy the file to each server/client so they can all find each other.

Children
  • Thanks Eric, this seems like an odd and disappointing thing to need to do in 2024.  Our system should not have any such issues as it is all running on one VM host right now, but if it is having some communication issues, I would like to get to the bottom of those as it may well be impacting on other things.

  • It's not a PlantSCADA problem. It's just a Windows workaround that should only be needed if your network is having trouble resolving computer names, which Dominic said he suspected was an issue with their old domain servers. I had to do that at one site which had deployment servers/clients on separate networks connected by a VPN but they had no domain controllers or DNS servers. On each local segment computers could resolve names, but not across the VPN to other segments.

  • Thanks Eric, indeed, if we were in some sort of unreliable environment, then firming everything up with some hard coding would be a serious consideration, however I think in our case, as it is still all running in our lab, there is something more fundamental that is going on.  Perhaps it is config at some level, I am not sure.

  • So I have now tried a hosts file on all of the VMs in the system, and it appears to be the same as before. I now have access to create a case with Aveva, so I Will do that.

  •  Hi  
    Nice to read from you.
    We cannot use domain controller to host PCS and other component from Plant SCADA. By reading  this coudl be his problem but we are mixing up.
    Mi suggestion is to get in contact with your local distributor and raise the case.

    I don't agree with the comment of  since the host file is only a solution to resolve DNS names. If the problem is accross domains we should have trusted domain (not necessary for all the plugs in that you want to use: AEL does, SMS and Deployment not). DNS suffix is a better and documented solutions for this.

    There are the SMC logs that contain a lot of info when you need to troubleshoot.

    It is something in your setup in my fair opinion so I will start to understand what it is timing up.

    Regards

  • Hi   
    The following fix will be available in Plant SCADA 2023 R2 U4 (October Update). Hope this update would resolve your issue.

    3455908 - Deployment server performance impacted on a domain environment without an internet connection.

    The deployment server configuration and its operation were slow when a client used a domain controller and both the client and domain controller were not connected to the internet. The Configurator will now launch quickly under these circumstances, and the deployment server configuration and deployment operations will respond promptly.

  • Thanks Jacky, I have gotten in touch with local Plant SCADA support (Schneider) and because of the other issues I was having on that machine, suggested KB41300 - and this somehow seems to have made a huge improvement.  I mean it fixed common services portal, which makes sense, but also somehow made configurator come up quicker and the deployment server service comes up quicker and the deployment activity has so far worked quicker and consistently. 

    ¯\_(ツ)_/¯

    I am now monitoring the situation.

    **Edit** deployment activity still times out, which makes more sense but unfortunately my problem persists, so hopefully it is the problem / fix you describe