(Editor’s note — an abridged version of this article was published in the June issue of CIO Review Magazine — see pp 17-18 of http://magazine.cioreview.com/June-2015/Datacenter/. What follows is the unabridged article.)

As technology managers for the County of San Mateo, we have fascinating jobs. On a day to day basis, the Information Services Department (ISD) is responsible for supporting a very large and varied organization – over 20 departments and approximately 6,000 employees — that provides a host of critical services such as public safety and health care to almost three-quarters of a million residents. Our geography and our population are both quite diverse. We are bordered by the Pacific Ocean to the west, the San Francisco Bay to the east, and the City of San Francisco to the north. More than half the county is unincorporated or open space, but we do have 20 incorporated cities and 23 school districts. Being in the heart of Silicon Valley (home to such technology leaders as Oracle, Facebook, and Genentech), we are a microcosm of the “digital divide,” with some of the wealthiest zip codes in the country neighboring some of the most underserved communities. We are also unique in that, unlike our neighboring counties, there is no one large city that dominates the populace or the politics.

To reach our long-term service goals, our CIO, Jon Walton, challenges us to not only understand the larger trends in technology but to apply those within the context and constraints of our mission as a government agency. Governments are often seen as being technology laggards, but our goal is to learn from the success of Silicon Valley and help transform government IT with similar leadership. We recognize that compared to our Silicon Valley private-sector brethren, we are designed to be more deliberate and risk averse because we utilize taxpayer dollars, conduct business in the open, and must serve all residents. As a government, we have the challenge of providing multiple channels of service including through traditional walk in traffic and telephone, but we also need to connect these traditional service solutions to more modern and efficient service delivery models.

In ISD we feel our role is to balance the need to continually serve our customers (the 20+ departments of the County government who, in turn, serve our residents) on a daily basis with the need to embrace technological trends that will, over time, make our services more efficient and connect government better with the residents it serves. “Connection” is the key word here; we have tied everything we do in ISD around these connections: connecting employees within the County government, connecting partner agencies to our systems and each other, and connecting residents to their government, all through infrastructure, data, and services.

The Need for a New Data Center for San Mateo County

Like many organizations (and certainly most public sector entities), we have many legacy systems and facilities, including those in our data center which were not originally designed for the scalability we need in 2015. Several years ago, we recognized that we were running out of physical space and tapping the maximum mechanical capability of our facility, including power and cooling. Simultaneously, our customer needs have been changing. As one could predict, there has been an increased demand for cloud-based applications, a more flexible service delivery model, and a desire to partner more closely with our local agencies. Additionally, the County has a strategic push around more transparent “Open Data” to enable all of our departments to engage residents in newer and more exciting ways. Unfortunately all of these goals could not be met quickly or cost effectively with the technology and facility constraints of our old data center.

So, in many ways, the technical forcing function to replace our aging data center infrastructure allowed us to start a new process that could be customer-centric and better align with the overall County mission and strategy. So, although we recognize that government has many differences from private business, we undertook a process similar to a way a private entity would design a new product or service model; starting with the customer needs. To do this, we assembled a team to start envisioning a plan. This was a cross-functional effort with members of our network, server, and project management groups as well as our departmental leadership and an overall program manager.

One of the additional challenges was that the County was planning for a new building which would be perfect for a new state-of-the-art data center. However, that wouldn’t be built for another several years in the future. Our technical needs and our customer requirements couldn’t wait. The team decided to relocate our main data center to a transitional facility – one designed to work for the next 3-5 years until we could move into the permanent space. Planning for a short-term space, in and of itself, created other challenges, one being to focus on a higher density design with a smaller footprint – approximately 1/5 the current size of 5,000 square feet – through virtualization of computing power, storage, network, and security. At the same time, we had to plan for scalability, such as allowing the environmental (heat removal in particular) and power to increase by up to 50% during this interim period. This smaller footprint will also facilitate a smoother future move to the permanent facility.

In researching short term locations, we looked at a number of co-location opportunities; additionally, we researched temporary space via container computing as well as traditional building lease situations. We discovered that due to the planning of the newer future building and a retrofit of an existing structure, we were going to have to not only support a Data Center move but also a personnel and equipment move. Specifically, our Radio Services team and a utility space had to be moved due to the construction and retrofit. This ultimately led us to get very creative by choosing a Data Center space that would not only allow us to house a secure and modern data center, but also address these other needed moves. We partnered with a co-location vendor that had excess space in their facility that hadn’t been built out for Data Center usage, and this partnership allowed us to solve our three issues with once space.

Customer Needs Driving Technology Decisions

The primary driver of our technology decisions was how we could meet our customer needs now and well into the future; even if our customers couldn’t anticipate their own needs. We partnered with our customers in the data center design process, working with them to truly understand how much storage they think they would need, what applications will require local hosting vs. which will migrate to the cloud, and what kind of connected devices they may be using in the future. This process was a pleasant surprise for many of our customers as historically IT designed infrastructure in the background or dictated the level of support the customer could have. With over 20 departments at the County, there are a lot of specialized solutions (e.g. the Controller’s office has very different needs from the hospital, etc.). Our team had to both ensure there would be support for these specialized solutions but find common components to see if there existed solutions that can be provided more efficiently.

For example, we have multiple agencies that share a public safety role (Sheriff, Office of Emergency Services, Probation, Public Safety Communications) – these departments share a common immediate communications infrastructure (Motorola APCO Project 25 700 Megahertz interoperable radio system), but also have common computational needs that are served via our CJI (Criminal Justice Interface) system. We worked with them to ensure that the departments had the freedom to choose best-in-breed systems, while at the same time allowing interoperability via an open interface system that is operated by ISD.

Given that the County is moving toward more cloud-based applications but will still retain many of its own on-premise applications, our new data center takes a hybrid approach. We created a private cloud with a converged infrastructure, utilizing technology to build a scalable, modular compute/storage infrastructure. This allows us to grow incrementally without headaches associated with other compartmentalized compute and storage products. In addition, other key applications were moved into an externally hosted environment, such as WorkDay, Office365, and ServiceNow. We retained a support model that would best suit customer needs – many departments would have both hosted and on-premise applications. Our converged infrastructure allows us to either provision virtual servers in our private cloud (instead of building physical servers) or use hosted services depending on the need.

We also went through the normal process of facility site selection and tiering of the data center based on what could work in our temporary facility. In analyzing in particular the mechanical infrastructure (cooling, power, UPS, etc.), we examined costs vs. benefits vs. time to implement, all within the context of keeping pace with the industry cycle of equipment life and upgrades.

The Other Side of the Coin – The Network

Building a new data center could not be done as a standalone project. The nature of what we were doing required more robust connectivity and redundancy within our LAN, our WAN, our second main data center, and our connection to the Internet. The county had been constrained by a low bandwidth connection to the Internet. Recently it had been modified to support multiple redundancies, but the connection speeds were still becoming quickly inadequate to support today’s robust cloud-based requirements. A separate effort in upgrading the network then became critical not only to the success of our customers interested and utilizing cloud solutions, but for our own inter-site needs in supporting a new Data Center that was not going to be housing the central connection point to the internet.

Therefore, the new data center had to be an integral part of the County’s significant network upgrade so as to leverage this new data center design. As our infrastructure, including the data center, became more distributed, it made our network that much more important. We had to architect our network in conjunction with the data center to create a more robust and redundant transport layer for WAN and Internet access, build a Software Defined Network, and better position the new facilities for high availability and disaster recovery across our WAN. The Internet upgrade that is part of our new network upgrade program would of course increase current speed and reliability for all users, but was also essential for us to retain the flexibility to do hybrid cloud solutions.

The enhanced network will enable us to build a private cloud and choose the best converged infrastructure components, including computing power, storage, network, security, and virtualization. This “pool of virtual resources” creates a set of IT services for needed demand – the ability to create elasticity that changes (stretches or shrinks) with customer needs. One of the most immediate benefits for our customers is our ability to provision virtual servers and storage to customers within minutes.

Our data center design also requires significant redundancy both within the converged infrastructure and also across our data centers (we retain secondary and ancillary data centers in other parts of the County) – our improved network design will enable a higher level of availability and business continuity.

The Current and Future Benefits

When we move into our new data center later this year, we will be able to meet our customer needs better than ever before, create flexibility over the coming years, and set us up for the next transition after that period. Fundamentally, in addition to increasing availability and performance, the new data center design is about faster provisioning of IT services to our clients. In fact, a new planned self-service portal will allow customers to both provision services for themselves (such as virtual servers, storage capacity, etc.) and have access to infrastructure monitoring systems to see performance and get automated alerts. Additionally, we project a significant financial benefit as well through cost reduction and avoidance going forward, particularly in the areas of lower power usage, less maintenance and administration, lower cooling costs, smaller physical space needed, and reduced hardware costs (both initial capital costs as well as refresh costs).

Going forward, we plan to leverage this new data center to also improve disaster recovery. This will be done through a combination of working more in the cloud as well as partnering with other government agencies to provide alternate routes and pathways to the Internet or to disaster recovery sites. For example, San Mateo County is working with the City and County of San Francisco to allow each agency to go through each other’s fiber network and Internet access points in the case of a major outage or disaster. There is a further opportunity for other local public agencies within our county to leverage the County’s dark fiber for similar purposes.

Although government has traditionally been a laggard in the adoption of new technologies and the embrace of new trends, we know we are uniquely positioned in Silicon Valley to leverage the expertise around us to both better serve the public and to be more efficient with taxpayer dollars in the process. All of this will come from the relentless focus on the notion of “connections” – on a technical level, a physical level, and even an interpersonal level. Fundamentally, that’s what technology is good at – it creates connections among people and between people and information, and it allows us to continually build and improve them. Our data center is solely just one instantiation of that notion.