This is Part 2 of a 5-part series on Evernote’s transition to Google Cloud Platform (GCP). To read this series from the beginning, see Part 1.
Our security team’s charter is to protect customer data. When we started the project to move to a cloud infrastructure, and Google specifically, we started work in parallel to understand how to keep customer data secure as we migrated into GCP. We focused on answering two main questions:
- Since we are going to be extending trust to Google to be the custodians of our data, do they have mature enough security controls that they won’t be adding risk to our service?
- Does GCP give us equivalent or better security controls that we can use to secure customer data?
Vendor Trust
We have an internal vendor review process that includes both our Legal and Security teams. Together, we review the Privacy and Security risks associated with using a new service provider to help us deliver features and functionality in the Evernote service.
When we review a vendor, we look at over 60 different controls that pertain to security and privacy. These areas align with how we’ve structured our internal programs and through this review we discover if, and where, a vendor diverges from our expectations. At a high level, most of these areas can be rolled up into:
- Organizational controls
- Infrastructure security
- Product security
- Physical security
- Data use related to privacy
We worked with Google to review their audit reports and answer additional technical questions. We found that in all areas, they met our expectations.
Next, we focused our evaluation efforts on whether they give us the controls we need to secure customer data.
Cloud Security Controls
We started our review by looking at all of the controls we have in place to protect customer data in our existing infrastructure. These controls included protective capabilities like remote access VPNs with two-factor authentication and firewalls that allow us to perform traffic filtering. They also included a lot of physical security controls like a sound physical perimeter, biometric authentication, cameras, and alarm systems that protect against a physical data theft. We talk about most of these infrastructure security protections on our security page.
We constructed a matrix and started answering questions about how we were going to transform each of these from our datacenter into cloud technology. To give you a sampling, here are some of the areas we looked at:
- Perimeter network security (ingress/egress traffic filtering)
- Internal segmentation
- Multifactor authentication
- Transit encryption
- Vulnerability management
- Administrative identity and access control
- Forensic logging
- Intrusion detection capabilities
- Change monitoring
We also took into consideration that operating in a multi-tenant cloud environment would introduce new threat models. We now need to care about memory and storage reuse in a different way than we have in the past. We also need to consider threats from other customers on the same hypervisor. Fortunately, Google has considered these threat models and discusses how they address some of them in a blog article.
For most of our controls we found an equivalent, cloud platform version. For data encryption at rest, we gained a security control that we hadn’t engineered on our own. For some controls, like IP whitelisting, we had to adapt our security architecture to not rely on traditional network controls. We did this by using GCP service accounts with Google-managed keys.
GCP Service Accounts and How to Use Them Securely
When you move to the cloud, that static CIDR block you had disappears into a mix of static and ephemeral, public IPs. IP whitelisting becomes much more operationally expensive. And it doesn’t exist on any of Google’s other cloud platform services.
We architect our security controls to have at least two layers of security between the internet and customer data. In our previous architecture, we had a well-defined, network perimeter and we housed all of our internal services within that perimeter. These internal services used API keys to talk to each other. We stored and distributed those keys in a secure way, but realized that one could leak or be stolen. If that did happen, we’d still have a second layer of control since you wouldn’t be able to use that key outside of our production environment. Access to that production environment requires two-factor authentication.
In Google, every GCP service is an internet service and none of them have a customer-facing control to whitelist access to only machines in your Google Compute Engine (GCE) project. We needed to find a way to add another layer of security between a stolen API key and customer data.
We solved this through use of GCP service accounts. Every GCE project gets a default service account. Any instance you launch within GCE can impersonate that service account to access other services. Behind the scenes, Google manages a public/private key pair and automatically rotates these keys every 24 hours. They do the same thing for custom service accounts. You can create a custom service account for each machine role and configure your virtual instance settings to use the appropriate service account. Now, any application running on that virtual instance using the GCP software development kit (SDK), can make use of the built-in, Google managed and rotated keys.
Our operations engineers never have a need to access these keys. Because Google rotates these keys once a day automatically, the only practical way to get access to them is to infiltrate our infrastructure, something we currently have adequate controls to guard against.
Overall, we are confident in the cloud security platform we have in place today and will keep looking to extend that platform to further enhance security and stay ahead of the changing threat landscape.
In Part 3 we discuss The Evernote Architecture in GCP and our Technical Transformation.
If you have any followup questions please join us on the Evernote forums.
