Slow performance during user synchronization

[Neil A]

Hi,

We're using EPM 2007, and have found lately that changes to the content of a
user group, or to a category, cause the server to slow down dramatically.
The queue shows a "User Synchronization for Project Web Access App Root Site
and Project WSS Workspaces" job running. If the number of users involved is
large, this can take hours, and all activity is affected. Even getting the
queue to refresh takes minutes.

Has anyone else experienced this? Is there something that can be done to
improve the performance when doing bulk changes?

TIA

 

[Paul Conroy]

Avoid user sync during core business hours/app usage periods and from running
in parrallel to Cube Building or Reporting DB rebuild.

 

[Neil A]

Paul,

Thanks for the quick reply. So is this normal? I haven't seen any
discussions or blogs on the topic.

 

[Paul Conroy]

How many users are being sync'd? Is there a DC on the same LAN as Project
Server?

Also, make sure other instances of PS are not sync'ing at the same time.

I wouldn't expect it to take hours. Depending on the volume of users it
could certainly take tens of minutes.

 

[Neil A]

Paul,

Some changes to a security category might affect hundreds of users - those
take hours. I just removed 6 users from a group, and that took 4 minutes,
during which time performance was clearly degraded.

There is a DC on the same subnet as the servers.

At the moment, we have only 1 instance of Project Server in active use.

 

[Paul Conroy]

Have you got a deep nested RBS tree ? Do you make use of dynamic category
permissions ?

If so and you have a reasonably large number of projects and users then I
would expect these ops to take considerably longer, especially for users
towards the top of the RBS tree.

 

[Neil A]

Paul,

The RBS runs to 6 levels deep, but we no longer use it to control
permissions - it was too hard to maintain. We now use project ownership or
project team membership rules to restrict access for one category. Other
categories don't use dynamic rules.

We have a fairly large number of projects (100+) and resources (1100+), and
things do seem to have been gettig slower as the numbers increase.

Thanks for your help Paul - you've given me some things to think about and
try out.

 

[steve]

so when we change a permission and it has to sync with all 1500 projects and
1500 WSS Sites, it takes about two days.. Unbelievably slow, and all
indications are NOT pointing to hardware/network or AD or config issues.
There appears to be a rediculous amount of code being processed to make each
update, not to mention, users whose permissions are changed are effectively
denied access to pwa as the sync appears to strip the users, update the
permissions and add them back... Seriously, is this yet another ms feature..
??? 2 days...

 

[James Fraser]

so when we change a permission and it has to sync with all 1500 projects and
1500 WSS Sites, it takes about two days..  Unbelievably slow, and all
indications are NOT pointing to hardware/network or AD or config issues.  
There appears to be a rediculous amount of code being processed to make each
update, not to mention, users whose permissions are changed are effectively
denied access to pwa as the sync appears to strip the users, update the
permissions and add them back... Seriously, is this yet another ms feature..
???   2 days...  

I'm not certain, but I think there were some performance enhancements
on one of the recent releases for this. Maybe the infrastructure
update?

BUT, two days means you have a problem or really slow servers. In an
environment with 1000 SharePoint sites, 1000 users, 2000 projects,
we've seen permission changes take as long as maybe 45 minutes, pre
infrastructure update.

Are your permissions similar to the out of the box set up?


James Fraser

 

[Doug]

James, can you share what your environment looks like?

 

[James Fraser]

James,  can you share what your environment looks like? ....

As an EPM consulting shop, we host a number of instances on our
hardware, and also are exposed to many different client environments.

One of our larger instances is 1018 active users, (1549 total), 1386
projects, RBS that is four layers deep IIRC, and about half the
projects have sharepoint sites. 800 users connect to the system more
than once a week.

SQL server sees most of the load and is running SQL 2005 on an eight
core machine, 16 GB RAM. ( Quad core would likely be fine for this,
but we are going to be increasing the user count on this hardware.)

Application server is similar, but doesn't see near as much load. We
are also running the Timesheet Auto Status service _very_ frequently,
which generates CPU use.

Is that what you're looking for? We have other large instances with
similar stats. In one case, more users but fewer projects, and in
another, far fewer users but thousands of projects. In all of these
instances, the SQL server performance drives much of Project Server's
performance.

I can provide more details if you'd like, but I'm not sure what you're
curious about.


James Fraser

 

[steve]

Hi James, thanks for the response. We have two dual proc servers up front
(web and app), and a quad proc sql box, that ties into SAN via fiber. Im not
seeing a huge increase in proc or mem utilization, although, i have noticed
the AD auth with the DC is probably slower than it should be. most of the
transactions coming across the profiler occur in 0 to 3 miliseconds except
the auth.