This module covers qualities of the data engineering solution beyond functionality and performance. It addresses reliability, policies, and security. Designing for security and compliance. Security is a broad term, includes privacy, authentication and authorization, and identity and access management, it could include intrusion detection, attack mitigation, resilience, and recovery, so security really appears across the technologies, not in just one place. You need to be aware of the granularity of control for each service. The exercise is imagine there are two people, one needs access and the other must not have access. What's the smallest unit or degree of control the technology supports? Can you distinguish security to an individual field or to a row or a record, or to specific columns, or to a specific database or entity, or just the actions that can be performed on the service. Commit a security checklist to memory, sometimes just running down a list will rapidly identify a solution. A key concept is to assign roles to groups and use group membership to grant permissions to individuals. How will the service be monitored or reported and how often will these items be reviewed? Finally you need to know what logs and reporting are available from each technology. A policy is set on a resource and each policy contains a set of roles and role members. Resources inherit policies from parents, so a policy can be set on a resource, for example a service, and another policy can be set on a parent such as a project that contains that service. The final policy is the union of the parent policy and the resource policy. What happens when these two policies are in conflict? What if the policy on the resource only gives access to a single, lets say Cloud storage bucket and restricts access to all other buckets? However at the project level, a rule exist that grants access to all buckets in the project. Which rule wins? The more restrictive rule on the resource or the more general rule on the project. If the parent policy is less restrictive, it overrides a more restrictive resource policy, so in this case the project policy wins. Folders map well to organization structure. It's a way to isolate organizations or users or products while still having them share billing and corporate resources. There are many encryption options for data at rest and in storage. Default encryption at rest uses the key management system, KMS to generate KEK which are key encryption keys and DEKs, the data encryption keys. When you use Cloud Dataproc, cluster and job data is stored on persistent disks associated with the compute engine VMs in your cluster and in a Cloud storage bucket. This PD and bucket data is encrypted using a Google generated data encryption key, the DEK and key encryption key, the KEK. Customer managed encryption keys, CMEK is a feature that allows you to create, use, and revoke the key encryption key, the KEK, Google still controls the data encryption key, the DEK. Client-side encryption simply means that you encrypt the data or file before you upload it to the Cloud. Key concepts, Cloud armor, Cloud load balancing, Cloud firewall rules, service accounts, separation into front-end and back-end, isolation of resources using separate service accounts between services. Because of the pervasive availability of firewall rules, you don't have to install a router and a network at a particular location to get firewall protection. That means you can layer the firewalls as shown in this example. Because of pervasive support for service accounts you can lock down connections between components. When faced with a security question on an exam or in practice, determine which of the specific technologies or services is being discussed. For example, authentication and encryption. Then determine exactly what the goals are for sufficient security. Is it deterrence? Is it meeting a standard for compliance? Is the goal to eliminate a particular risk or vulnerability? This will help you define the scope of a solution, whether on an exam or in application.
