4-minute read

Navigating the complexities of Public Safety Power Shutoffs

Wildfires and extreme weather are a year-round risk in California, especially during fire season, generally September through December. Our client regularly monitors local fire danger and extreme weather conditions across the state. To help protect customers and communities during extreme weather events, the EOC may decide to shut off electric power to prevent electric systems or downed power lines from becoming the source of an ignition. This is called a Public Safety Power Shutoff (PSPS) activation. Not only do these events have a serious public safety impact, but they are also highly scrutinized by multiple regulatory bodies.

The EOC uses a suite of applications to make the decision to turn power off and back on, and they cannot afford any downtime during a PSPS activation. The availability of these applications is critical for public safety. Below are some examples of the applications’ functionality:

• Monitor weather and electrical circuit information to help make decisions on PSPS de-energizations during an EOC activation.
• Display current distribution electrical outages in the service territory.
• Monitor real-time air quality index to notify field staff when to use the appropriate ventilation mask if the safety threshold is exceeded.
• Communicate PSPS outage details to utility partners during fire season.

Addressing cloud dependency and network resiliency challenges

At the outset of our engagement, the EOC systems had a strict dependency on the uptime of a single region within a single cloud provider. Furthermore, our client relies on exchanges to connect to the cloud. Recently, their data center connection to the cloud failed when an exchange’s own path to the cloud provider failed. A natural disaster such as an earthquake could cause loss of connection as well.

The Active-Active multi-region / on-prem failover approach was born out of these challenges.

Developing the solution

We worked with the utility to deliver significant improvements to their EOC systems reliability and ultimately removed five single points of failure that the EOC systems rely on.

Leveraging the AWS Well Architected Framework, we centered our approach around Kubernetes, the cloud-native de facto standard to deploy and operate containerized applications. This approach, along with the myriad other AWS tools listed above, enables the deployment of software images across cloud providers while remaining serverless, thus avoiding traditional sys-ops patching and upgrading of operating systems.

Developers can now check in code to a single repository, and the pipeline builds, security-scans and deploys the updates to multiple cloud providers.

Using the AWS Resilience Hub, we assess, improve, test, and track the resiliency posture of each application to protect it from disruptions and improve overall business continuity.

Within this ongoing engagement, Logic20/20 has introduced the most up-to-date tools and approaches for software development and architecture. Our team keeps abreast of the rapidly changing ecosystem of cloud solutions and has the expertise and flexibility to change tools mid-project when warranted. We’re also working closely with our client’s in-house developers and architects, helping them upgrade their skillsets to encompass the latest cloud technologies.

Partnering for enhanced community safety

Over the years, our client has built a strong reputation on its commitment to preserving the safety of the communities it serves. In working with their emergency operations center to enhance the resiliency of its applications, Logic20/20 is helping them maintain that reputation. Together, we are taking the first steps towards the utility’s goal of maximizing availability for continuous, reliable emergency operations that will serve Californians today and for years to come.

Sean D. Quealy is a Manager in Logic20/20’s Digital Transformation practice.