In many servers, and even many desktops, storage space is wasted due to the necessities of growth, planning and disk capacity granularity. If we have twenty servers each with 300GB drive arrays but each only using 80GB of that capacity, we have large waste. With a SAN would could consolidate to just 1.6TB, plus a small amount necessary for overhead, and spend far less on physical disks than if each server was maintaining its own storage.
Once we begin consolidating storage, we begin to look for advanced consolidation opportunities. Having consolidated many server filesytems onto a single SAN, we have the chance, if our SAN implementation supports it, to deduplicate and compress that data which, in many cases, can result in significant utilization reduction. So the 1.6TB in our example above might actually end up being only 800GB or less. Suddenly, our consolidation numbers are getting better and better.
To efficiently leverage consolidation, it is necessary to have scale. And this is where SANs really shine — when the number of attaching nodes becomes very large. SANs are best suited to large scale storage consolidation. This is their sweet spot and what makes them nearly ubiquitous in large enterprises and very rare in small ones.
SANs are also very important for certain types of clustering and shared storage that requires single shared filesystem access. This is actually a pretty rare need outside of one special circumstance — databases. Most applications are happy to utilize any type of storage provided to them, but databases often require low-level block access to be able to properly manipulate their data most effectively. Because of this, they can rarely be used, or used effectively, on NAS or file servers. Providing high availability storage environments for database clusters has long been a key use case of SAN storage.
Outside of these two primary use cases, which justify the vast majority of SAN installations, SAN also provides for high levels of storage flexibility. Potentially, SANs can make it very simple to move, grow and modify storage in a large environment without needing to deal with physical moves or complicated procurement and provisioning. Again, like consolidation, this is an artifact of large scale.
In very large environments, SAN can also provide a point a demarcation between storage and system engineering teams, allowing there to be a handoff at the network layer, generally of fibre channel or iSCSI. This clear separation of duties can be critical in allowing for teams to be highly segregated in companies that want highly discrete storage, network and systems teams. This allows the storage team to do nothing but focus on storage, and the systems team to do nothing but focus on the systems without any need for knowledge of the other team's implementations.
For a long time, SANs also presented themselves as a convenient means to improve storage performance. This is not an intrinsic component of SAN, but an outgrowth of their common use for consolidation. Similar to virtualization when used as consolidation, shared SANs will have a natural advantage of having better utilization of available spindles, centralized caches and bigger hardware than the equivalent storage spread out among many individual servers. Like shared CPU resources, when the SAN is not receiving requests from multiple clients, it has the ability to dedicate all of its capacity to serving the requests of a single client. That provides an average performance potentially far higher than what an individual server would be able to affordably achieve on its own.
Using SANs for performance is rapidly fading from favor, however, because of the advent of SSD storage. SSDs with incredibly low latency and high IOPS performance are dropping in price to the point where they are being added to stand alone servers as local cache or potentially even being used as mainline storage. As a result, the bottleneck of the SAN's networking becomes a larger and larger factor, making it increasingly difficult for the consolidation benefits of a SAN to offset the performance benefits of local SSDs. SSDs are potentially very disruptive for the shared storage market as they bring the performance advantage back towards local storage — just the latest in the ebb and flow of storage architecture design.
The most important aspect of SAN usage to remember is that SAN should not be a default starting point in storage planning. It is one of many technology choices and one that often does not fit the bill as intended — or does so but at an unnecessarily high price point either in monetary or complexity terms.
Instead, start by defining your business goals and needs. Select SAN when it solves those needs most effectively, but keep an open mind and consider the overall storage needs of the environment.