Good Morning!
Current Configuration:
1. 1 Central, Primary Site; 1 Child Primary Site; 4 Child Secondary Sites (children of the Child Primary Site)
2. The Primary Site servers have the SQL server databases roles as well, so there are no connectivity issues between SQL and SCCM per site.
3. The Central Site (CEN), was recently upgraded from Server 2003 to Server 2008 R2 and the SCCM environment was migrated using a side-by-side migration.
4. We ran into several issues with the server migration, all of which have been remedied, EXCEPT for this particular issue.
5. I am not sure if this issue is a by-product of a larger replication issue, but I wouldn't put it past it.
Here is what we have done to troubleshoot the issue:
1. Ran a site reset (reset permissions to ensure the accounts and servers have the adequate permissions to the SQL server database.)
2. Examined the CI_CIAssignments table on the SQL database on CEN (where the error is occuring) to ensure all CI Assignments exist.
3. For purely informative purposes: The CI_CIAssignments table is directly-related to the Software Updates --> Deployment Management SCCM node. The Deployment ID attribute are equvialent to the rows from the Assignment_ID column in the CI_CIAssignments table within SQL. I have verified between the two Primary Site databases that the entries are in sync; HOWEVER, there are some stale records that do not exist on the Primary Child Site. My assumption is that this would be the case if, say, there were only one Primary Site server and Deployment Management objects were created/removed and then later a second Primary Site were brought online. The objects that were created/removed prior to the second primary site server being brought online would never replicate from the parent primary site server as they are no longer relevant or in use.
Long story short, we are receiving the following error CIAMGR.LOG:
Wait timed out after 11 minutes while waiting for at least one trigger event. SMS_CI_ASSIGNMENT_MANAGER 9/16/2013 10:00:54 AM 8312 (0x2078) ProcessReplicatedFiles. SMS_CI_ASSIGNMENT_MANAGER 9/16/2013 10:00:59 AM 8312 (0x2078) ProcessReplicatedFilesInRetry. SMS_CI_ASSIGNMENT_MANAGER 9/16/2013 10:00:59 AM 8312 (0x2078) ProcessChangedObjects. SMS_CI_ASSIGNMENT_MANAGER 9/16/2013 10:00:59 AM 8312 (0x2078) Failed to process CI Assignment changes, Status = SQL ERROR, changes will be retried on next processing cycle. SMS_CI_ASSIGNMENT_MANAGER 9/16/2013 10:00:59 AM 8312 (0x2078) STATMSG: ID=5804 SEV=E LEV=M SOURCE="SMS Server" COMP="SMS_CI_ASSIGNMENT_MANAGER" SYS=<CENTRAL SITE SERVER NAME> SITE=<CENTRAL SITE CODE> PID=476 TID=8312 GMTDATE=Mon Sep 16 14:00:59.320 2013 ISTR0="" ISTR1="" ISTR2="" ISTR3="" ISTR4="" ISTR5="" ISTR6="" ISTR7="" ISTR8="" ISTR9="" NUMATTRS=0 SMS_CI_ASSIGNMENT_MANAGER 9/16/2013 10:00:59 AM 8312 (0x2078) ProcessObjectsInRetry. SMS_CI_ASSIGNMENT_MANAGER 9/16/2013 10:00:59 AM 8312 (0x2078) ProcessAffectedObjects. SMS_CI_ASSIGNMENT_MANAGER 9/16/2013 10:00:59 AM 8312 (0x2078) HandleSiteAttach. SMS_CI_ASSIGNMENT_MANAGER 9/16/2013 10:00:59 AM 8312 (0x2078) Process CI Assignment Delete notification files. SMS_CI_ASSIGNMENT_MANAGER 9/16/2013 10:00:59 AM 8312 (0x2078) WaitForEvent. SMS_CI_ASSIGNMENT_MANAGER 9/16/2013 10:00:59 AM 8312 (0x2078) Waiting for changes for 29 minutes SMS_CI_ASSIGNMENT_MANAGER 9/16/2013 10:00:59 AM 8312 (0x2078)
This occurs when there are *.CIA objects in the ciamgr.box inbox. All other objects, such as *.CIN files, process normally. For example, the below snippet is from the CIAMGR.LOG as well after a single attribute (such as custom severity) is altered on a Deployment Management object.
Processing CI Change Notification file D:\SMS\inboxes\ciamgr.box\98624.CIN. SMS_CI_ASSIGNMENT_MANAGER 9/16/2013 9:25:16 AM 8312 (0x2078)
************Processing CI Assignment {B6BCD488-56E4-4E7D-A438-15E7C00FE81F}************ SMS_CI_ASSIGNMENT_MANAGER 9/16/2013 9:25:16 AM 8312 (0x2078)
Successfully updated Policy for CI Assignment {B6BCD488-56E4-4E7D-A438-15E7C00FE81F} SMS_CI_ASSIGNMENT_MANAGER 9/16/2013 9:25:16 AM 8312 (0x2078)
Successfully updated Policy Targeting for CI Assignment {B6BCD488-56E4-4E7D-A438-15E7C00FE81F} SMS_CI_ASSIGNMENT_MANAGER 9/16/2013 9:25:16 AM 8312 (0x2078)
Successfully updated CRCs for CI Assignment {B6BCD488-56E4-4E7D-A438-15E7C00FE81F} SMS_CI_ASSIGNMENT_MANAGER 9/16/2013 9:25:16 AM 8312 (0x2078)
STATMSG: ID=5801 SEV=I LEV=M SOURCE="SMS Server" COMP="SMS_CI_ASSIGNMENT_MANAGER" SYS=W31SVIRSMSCNT01 SITE=NRO PID=476 TID=8312 GMTDATE=Mon Sep 16 13:25:16.699 2013 ISTR0="MS10 - Servers" ISTR1="" ISTR2="" ISTR3="" ISTR4="" ISTR5="" ISTR6="" ISTR7="" ISTR8="" ISTR9="" NUMATTRS=1 AID0=414 AVAL0="{B6BCD488-56E4-4E7D-A438-15E7C00FE81F}" SMS_CI_ASSIGNMENT_MANAGER 9/16/2013 9:25:16 AM 8312 (0x2078)
************Successfully processed CI Assignment {B6BCD488-56E4-4E7D-A438-15E7C00FE81F}************ SMS_CI_ASSIGNMENT_MANAGER 9/16/2013 9:25:16 AM 8312 (0x2078)
Successfully processed CI change file D:\SMS\inboxes\ciamgr.box\98624.CIN. Deleting it. SMS_CI_ASSIGNMENT_MANAGER 9/16/2013 9:25:16 AM 8312 (0x2078)Once those *.CIN files are processed and the CIAMGR.LOG goes back to standard processing changes every 30 minutes, we receive the error posted above (Failed to process CI Assignement Changes - SQL ERROR). This occursregardless of the existence of *.CIA manager files in the CIAMGR.BOX inbox. You can force changes by dropping files in the box but the same result occurs (SQL ERROR).
Does anyone have any insight into what the generic "SQL ERROR" error can indicate. My assumption is this is a SQL backend issue, but I don't know what to look for as far as corrupt rows or columns are concerned.
Ultimately this appears to be impacting the Software Update synchronizations between the two Primary Site servers. When a synchronization is run on the Primary Site server, the child site server Deployment Management objects are not being sent the updates for the various objects. If we manually change something (such as the Custom Severity attribute) on an update that is part of the Deployment Management object, this can force a *.CIN change, but not the *.CIA object change.
Any insight or assistance is greatly appreciated!
Thank you!