I’ve already received a few comments and mails from people asking questions…
One comment I’ve received is about LINQ’s IN operator(method?) and SQL Server plan cache pollution. I’m researching the issue a bit before I post a complete answer, but I’ll start with some background information on memory management in SQL Server. I co-authored a paper on a new memory management algorithm that I helped develop that went into SQL Server 2005. I don’t think it has changed much in SQL Server 2008 from what I’ve seen of the November CTP at least.
The interesting thing about memory in SQL Server is that, unlike some of the other vendors in the database space, SQL Server runs all of its users within the same server process. (This derives from UNIX traditionally using fork() to create another server process to service each user). Windows has better thread support, and as a result it was possible to make SQL Server run all of the user activities within a single process. This has advantages because you can avoid inter-process communication for a lot of things, like a plan cache. You can also “be smart” on things like which activity to schedule next in ways that an Operating System will not do. Often this can lead to large performance gains.
One downside of this single-process approach is that, at least on 32-bit machines, virtual memory space becomes a limited resource, especially on larger deployments. Every component has to fit into about 1.4GB of space (with the remainder taken up by stacks for threads, address space for mapping DLLs into the process, etc). The limitation does not exist on 64-bit machines, which is why I recommend that all new serious SQL Server deployments closely consider their architecture when purchasing new machines.
This paper was accepted at the Conference on Innovative Data Systems Research (CIDR), of which Jim Gray was one of the founding organizers. It has some details in the kinds of problems that were solved and the performance gains that were achieved by more actively managing the memory consumption and consumption rates of each internal subcomponent. In a sense, this is how one could build a special-purpose memory management system for a machine that only ran a database.
Memory-CIDR2007Accepted2.doc (400 KB)
I will start working on some posts on some of the interesting things that happen in the query processor to handle IN clauses (in SQL, not LINQ). Then I’ll do some work with LINQ’s IN equivalent and see what interesting things I can post.
Thanks,
Conor Cunningham
The post User Question: Plan Cache Pollution appeared first on Conor Cunningham.