Google Cloud Platform (GCP) continues to expand its offerings and remains a top-tier cloud infrastructure service provider. It offers a wide range of services, including Compute Engine (VMs), Google Kubernetes Engine (GKE), BigQuery, Cloud Storage, and Cloud AI Platform. However, in recent years, several vulnerabilities have been reported in some of GCP's services, such as privilege escalation in GKE and CloudSQL.
As the adoption of GCP grows and more organizations choose it as their cloud infrastructure, it is essential to understand how to conduct forensic investigations and analyze logs in GCP. A key aspect of any investigation is to determine the principals and identities involved in the incident.
In this blog post, we'll explain how to identify principles and understand the identities present in GCP logs, including impersonation and third-party identities.
The first part of this two-part blog series will cover the logging system in Google Cloud Platform (GCP) and its fields. We will also provide a comprehensive overview of the various identity types in GCP, along with a detailed explanation of service accounts and impersonations within the platform.
- Overview of GCP Logs
- Identity Types in GCP: the Different Identities Available
- authenticationInfo in GCP Logs: Understanding Who, What, and Where
- Service Accounts: An Introduction
- How to Use Service Accounts: Authentication and Impersonation
Overview of GCP Logs
GCP has several logging categories, such as Cloud Audit Logs, Service Logs, Agent Logs and Network Logs.
Cloud Audit Logs:
Audit logs answer the question many various cloud audit logs answer and that is “who did what, where, and when?”
- Admin Activity: contain log entries for API calls of configuration changes or changes to resource metadata.
- System Event: contain log entries of actions altering resource configurations, specifically non-human administrative actions, generated by Google systems.
- Data Access: contain log entries of API calls that access resource configuration or metadata, along with user-initiated calls that create, modify, or read user-provided resource data.
- Access Transparency: contain log entries of actions taken by Google personnel when accessing your content.
Service Logs:
Service logs are logs that developers can create for deploying code to Google Cloud. For example, when deploying a web application to Google App Engine, logging statements in the application code are captured and sent to Cloud Logging.
Agent Logs:
Agent logs in Google Cloud Platform (GCP) are logs generated by agents running on virtual machines, like the Ops Agent or legacy Cloud Logging agent (fluentd), which collect and send data from known file locations or logging services to Cloud Logging for monitoring and management.
Network Logs:
Network logs provide detailed telemetry for network and security operations and can be listed in these logs:
- VPC Flow: contain entries of VPC network flows and can be used for network monitoring, security analysis and forensic investigations related to attacks such as Denial of Service (DoS), Port Scanning, Reconnaissance, Data Exfiltration, and other security incidents.
- NAT Gateway: contain information on NAT network connections and errors.
- Firewall Rules: contain logs that track and analyze the effects of firewall rule on network traffic.
With a better understanding of the available logs in GCP, this blog post will focus on the Audit Logs, specifically the Admin Activity and Data Access logs.
Identity Types in GCP: the Different Identities Available
Before discussing logging that assists forensic investigators in identifying the principals behind an operation, it's important to understand and map out the entities existing in GCP.
In GCP, four main entities operate. The following sections will discuss each of these identities individually. Here's a brief overview:
- User: A user account that interacts with GCP resources(human users). Typically authenticated using Google accounts or other identity providers integrated with Google Cloud Identity and Access Management (IAM).
- Service Accounts: Accounts intended for non-human entities, like applications and virtual machines to access Google Cloud services programmatically. Authenticate with a private/public key pair.
- Service Agents: - Google-managed service accounts used by some Google Cloud services to access user resources. These service agents are created by Google when necessary and are visible in a project's allow policy and audit logs, but not in the service account section.
- Workload Identities: - Used in Google Kubernetes Engine (GKE) for workloads to authenticate and authorize with other Google services without manual management of service account keys. They can also be used with other Identity Providers (IDPs) and cloud providers like AWS (Amazon Web Services) for secure cross-platform communication.
AuthenticationInfo in GCP Logs: Understanding Who, What, and Where
Audit logs, which include Admin Activity and Data Access logs, record the identity of the entity performing operations on a Google Cloud resource. The caller's identity is stored in the authenticationInfo field of the AuditLog objects.
The AuthenticationInfo field contains details such as the authentication method used and the email address or service account key associated with the caller, providing details about the authentication context for the logged operation.
To understand the importance of the fields of the authenticationInfo field, it would be beneficial to understand its structure and fields. Google’s documentation provides detailed information on this.
An example of a full possible occurrence of an authenticationInfo:
A description of the fields (from Google’s documentation):
It’s apparent that there are two additional structs in the authenticationInfo field: serviceAccountDelegationInfo and thirdPartyPrincipal.
The serviceAccountDelegationInfo is related to service account delegation which will be covered later on in this blog post.
thirdPartyPrincipal is used to track authentication from entities outside of Google's services, such as external identity providers (if applicable).
An example of a serviceAccountDelegationInfo:
A description of the fields (from Google’s documentation):
A deeper look into the firstPartyPrincipal and thirdPartyPrincipal:
After examining the authenticationInfo field in detail, the next step is to review the implementation of Service Accounts.
Service Accounts: An Introduction
As mentioned, one of the entities in GCP is Service Accounts.
Service Accounts provides an identity for carrying out service to service interactions in a project without supplying credentials.
An example usage of a service account can be writing an application running on a Compute Engine VM instance, and said application needs to read a file from Cloud Storage. A service account with the correct role (set of permissions, given the write scope) can be assigned to the instance. Then the application running on the instance would be able to use the service account without embedding any secret keys or credentials in the instance or application code.
Service accounts can be identified with a Google-managed email with the prefix of gserviceaccount.com.
There are three types of service accounts:
- User created (custom)
- Built-in service accounts
- Google managed
User created (custom):
Users can create a service account to meet their specific needs. When managing a service account, users must provide user-managed keypairs for authentication, with Google storing only the public portion.
Up to 10 user-managed service account keys per service can be createdand managed using the IAM API, console, or gcloud.
These service accounts are considered as user managed service accounts.
Built-in service accounts:
Built-in service accounts which are a subset of user managed service accounts, are created automatically when enabling certain Google Cloud services. These service accounts have key-pairs managed by Google for both the private and public keys. The public key can be used for signing for up to two weeks, while the private key is securely held in escrow and is not directly accessible.
The Default Compute Engine service account is an example of a Built-in service account. It is automatically created for each project with an auto-generated name and email address, such as 3xxxxxxx5-compute@developer.gserviceaccount.com. This service account is automatically assigned the Editor role for the project and is enabled by default on all instances created using gcloud or the console. When starting a new instance with gcloud, the default service account is automatically enabled. Changing this default setting is possible by specifying a different service account or disabling service accounts for the instance.
Google managed:
Google-managed service accounts, like the Google APIs Service Agent (identified as <project_number>@cloudservices.gserviceaccount.com),are created and managed by Google for service management purposes. They are commonly used to support service functionality (and additionally run operations on behalf of services or users). These accounts, also known as service agents, are visible in IAM policies and audit logs but are not listed in the service account section. We will explore these accounts further in the next part of the blog.
How to Use Service Accounts: Authentication, Impersonation and How to Find Them in the Logs
Using a service account can be achieved by either activating the service account or impersonating it.
Service Account Activation:
When activating a service account with gcloud, you authorize gcloud and other tools in the Google Cloud CLI to use the service account's credentials for making requests to Google Cloud services. This activation process imports the service account's credentials from a file containing a private authorization key (JSON or p12) and activates them for use with gcloud.
The activation imports the service accounts credentials from a file containing a private authorization key (JSON or p12) and activates it for gcloud in this case.
Activating a service account is like logging in with a user account (using gcloud auth login) but using a service account instead.
An example of an activation in gcloud:
gcloud auth activate-service-account <SERVICE_ACCOUNT>@<DOMAIN>.COM --key-file=<path_to_key_file> --project=<PROJECT_ID>Service Account Impersonation:
Impersonation for service accounts allows users to delegate permissions for a service account to act on behalf of another user or service account. This enables the service account to access resources and perform actions requiring specific permissions usually associated with the user or service account being impersonated. It enables secure usage of applications or users, ensuring they have the necessary permissions without revealing sensitive information or requiring significant code modifications.
Some of Google’s documentation on impersonation:
“When an authenticated principal, such as a user or another service account, authenticates as a service account to gain the service account's permissions, it's called impersonating the service account. Impersonating a service account lets an authenticated principal access whatever the service account can access. Only authenticated principals with the appropriate permissions can impersonate service accounts.
Impersonation is useful when you want to change a user's permissions without changing your Identity and Access Management (IAM) policies. For example, you can use impersonation to temporarily grant a user elevated access, or to test whether a specific set of permissions is sufficient for a task. You can also use impersonation to locally develop applications that can only run as a service account, or to authenticate applications that run outside of Google Cloud.”
Why impersonate?
Legitimate service account impersonations in GCP can have use cases such as:
- Delegated access: When a service account needs to access resources that it wouldn't normally have access to, such as when a service account needs to access resources in a different project.
- Automation: Service accounts are used to automate tasks, and impersonation allows one service account to perform actions on behalf of another, simplifying access management.
- Access to external services: A service account might need to access external services that require a specific identity for authentication purposes.
On the other hand, malicious actors might use impersonation for unauthorized access to resources or to bypass security controls. Some examples of malicious use cases include:
- Data exfiltration: A malicious actor might impersonate a service account to access and exfiltrate sensitive data.
- Privilege escalation: Impersonation can be used to escalate privileges and gain access to resources that are restricted to higher privileged accounts.
- Covering tracks: Impersonation can be used to perform malicious activities while masking the identity of the attacker, making it harder to trace the actions back to the original source.
Impersonation can take two forms:
- Impersonation through service activation
- Impersonation on a specific invocation
Impersonation through service account activation
This involves logging in with a service account (and activating it) using credentials from a file as mentioned earlier. It allows gcloud to use the service account credentials for requests, effectively setting the environment to that of the service account we logged in as.
An example of such an impersonation:
gcloud auth activate-service-account --key-file= <path_to_key_file> --impersonate-service-account=<SERVICE_ACCOUNT_EMAIL>
Impersonation through service account activation: How to identify it in the logs?
Under this specific impersonation, the logs will display the impersonated service account as the principalEmail in the authenticationInfo field. Additionally, the serviceAccountKeyName in the logs indicates authentication was done using a key for activation.
The credentials loaded can be revoked using gcloud auth revoke.
Impersonation on a specific invocation
Specific invocation impersonation enables impersonation for a specific action, granting temporary access to resources as the impersonated service account without impacting the broader environment. This approach allows for impersonation without the need to create or activate a key. Additionally, it supports creating a chain of service accounts for delegation chain impersonation.
To perform this kind of impersonation, a user or service account must have the necessary permissions. These permissions are listed in a role named Service Account Token Creator, which includes the following permissions:
iam.serviceAccounts.get
iam.serviceAccounts.getAccessToken
iam.serviceAccounts.getOpenIdToken
iam.serviceAccounts.implicitDelegation
iam.serviceAccounts.list
iam.serviceAccounts.signBlob
iam.serviceAccounts.signJwt
resourcemanager.projects.get
resourcemanager.projects.listThese permissions are necessary for impersonation because the account initiating the impersonation uses the GenerateAccessToken method to obtain a temporary token for the gcloud operation they wish to perform.
The user who generated the access token is specified in authenticationInfo.PrincipalEmail, along with details of the service account for which the token was created, found in request.name and resource.labels.email_id.
A little about impersonation with delegation chains
Some information provided from Google’s documentation:
“For this gcloud invocation, all API requests will be made as the given service account or target service account in an impersonation delegation chain instead of the currently selected account. You can specify either a single service account as the impersonator, or a comma-separated list of service accounts to create an impersonation delegation chain. The impersonation is done without needing to create, download, and activate a key for the service account or accounts.
To make API requests as a service account, your currently selected account must have an IAM role that includes the iam.serviceAccounts.getAccessToken permission for the service account or accounts.
The roles/iam.serviceAccountTokenCreator role has the iam.serviceAccounts.getAccessToken permission. You can also create a custom role.
You can specify a list of service accounts, separated with commas. This creates an impersonation delegation chain in which each service account delegates its permissions to the next service account in the chain. Each service account in the list must have the roles/iam.serviceAccountTokenCreator role on the next service account in the list. For example, when --impersonate-service-account= SERVICE_ACCOUNT_1,SERVICE_ACCOUNT_2, the active account must have the roles/iam.serviceAccountTokenCreator role on SERVICE_ACCOUNT_1, which must have the roles/iam.serviceAccountTokenCreator role on SERVICE_ACCOUNT_2. SERVICE_ACCOUNT_1 is the impersonated service account and SERVICE_ACCOUNT_2 is the delegate.”
An example command using an impersonation with a delegation chain:
gcloud storage buckets list --impersonate-service-account=[SA_email](or comma separated emails -> SA1,SA2,SA3)In delegation chains with multiple service accounts, each entity in the chain generates an access token using GenerateAccessToken. The chain is shown in metadata.identityDelegationChain as an array, with each entity creating a token resulting in a shorter chain.
When executing the gcloud operation within the impersonation chain, the operation appears as if it was performed by the last service account in the chain. However, the entire delegation chain can be viewed in authenticationInfo.serviceAccountDelegationInfo.[<item location>].firstPartyPrincipal.principalEmail, with the final object indicating the original user who initiated the impersonation chain (essentially, the chain in reverse).
Authorization levels when impersonating
When impersonating a service account using GenerateAccessToken, two authorization levels can be observed based on the delegation chain. For single impersonations, the permission scope is iam.serviceAccounts.getAccessToken:
.png)
When impersonating with a longer delegation chain, the permission scope is iam.serviceAccounts.implicitDelegation:
Default Compute Engine service account delegation
As previously mentioned, Google Compute Engine instances use the Default Compute Engine service account by default. However, it's possible to attach a different service account to a VM. When a specific service account is attached to a VM, operations from that VM will be seen as delegated operations originating from the Default Compute Engine service account. The service account attached to the VM can be identified in authenticationInfo.principalEmail, while the delegation from the Default Compute Engine service account can be found in authenticationInfo.serviceAccountDelegationInfo.firstPartyPrincipal.principalEmail.
Summary
In this first part of our two-part blog series, we've covered the basics of GCP logs for forensic investigations, the different identities in GCP, and a detailed exploration of how service accounts are used, impersonated, and how to analyze log components.
The second part will explore the remaining identities and instances where GCP does not log identities.
