Integrating Google Cloud Audit Logs: Beyond Turnkey Setup

The Google SecOps onboarding flow makes Cloud Audit Log ingestion look like a one-toggle problem. Click Direct Ingestion, point it at your org node, watch the events roll in, mark the ticket done. In a lab, this works. In a regulated, multi-thousand-project GCP estate, this is how you spend the next quarter explaining your ingestion bill and fielding “why is this detection so noisy?” tickets.

The hard part isn’t the integration; it’s deciding, with intent, which audit events deserve a seat in your SIEM and which ones you’re paying to store and never query.

I’ve built this pattern dozens of times across financial services, healthcare, and other heavily regulated estates. The shape of the problem is always the same: enabling Data Access logging is non-optional for compliance, but routing it raw into SecOps will saturate the pipeline with reads no analyst will ever look at. The fix is a deliberately designed filter on the way out of GCP, plus a second line of defense between GCP and SecOps for the events you couldn’t filter cleanly at the source.

What you’re actually subscribing to

“Cloud Audit Logs” is a category, not a stream. Five sub-streams sit underneath it, and each behaves differently:

Log type	Default state	Volume profile	Detection value
Admin Activity	On, can’t disable	Low to medium	High. The spine of GCP detection.
Policy Denied	On for most services	Low to medium	High. Denied attempts are loud signal.
Access Transparency	Eligible support tiers only; off otherwise	Very low when on	High. Google personnel access to your data. Compliance-relevant in regulated estates.
System Event	On, can’t disable	Low	Medium. GCP-side automated changes.
Data Access	Off by default for most services	Catastrophic when on	Mixed. Admin reads are gold; bulk object reads are noise.

One scoping note before going further. Cloud Audit Logs are one log category among many that SecOps can ingest from GCP via the Direct Ingestion sink. VPC flow logs (compute.googleapis.com/vpc_flows), NAT flows (compute.googleapis.com/nat_flows), DNS queries (dns.googleapis.com/dns_queries), firewall logs (compute.googleapis.com/firewall), load balancer requests (loadbalancing.googleapis.com/requests), Cloud IDS (ids.googleapis.com/threat, ids.googleapis.com/traffic), and a handful of others each have their own log_id(), their own volume profile, and their own filter discipline worth a dedicated treatment. The design pattern this post argues for (inclusion as default, surgical exclusions, layered defense at the sink and at the OTEL - OpenTelemetry - transform) applies to all of them, but the specifics differ enough that mixing them into one post would dilute both halves. This one stays in the cloudaudit.googleapis.com/* lane because audit logs are where most teams get stung first.

The first four you ingest in full, no apologies. They’re cheap, low-volume, and most detections worth writing depend on at least one of them. Admin Activity alone covers identity changes, IAM policy mutations, resource creation and deletion, and most of what a competent threat actor would need to touch.

Data Access is the one that ruins quarters. The minute you turn it on across all services (and many compliance frameworks read like they’re requiring exactly that), you’re capturing every storage.objects.get, every bigquery.jobs.create triggered by a backfill, every metadata read from every Cloud Run revision. In a 4,000-project estate where data engineering teams are paginating through Cloud Storage from automation service accounts every minute, the volume math becomes unfriendly very quickly.

(I’ve watched a single data pipeline produce more Data Access events in a day than the rest of the org combined for a week. It happens.)

The architecture choice

Three patterns dominate the way GCP Cloud Audit Logs reach Google SecOps:

Native direct ingestion is the default recommendation, full stop. It’s GCP-to-SecOps over Google’s own infrastructure: no Pub/Sub topic to size, no forwarder fleet to keep alive, no third-party SLA in the path. The exceptions are real but narrow. Heterogeneous SIEM destinations (you want the same logs in two places). Heavy enrichment requirements. Compliance constraints that demand an organizational data plane in between. Pub/Sub-to-Cribl-to-SecOps still has its place. It just isn’t the default anymore.

The piece that changes the calculus is Data Pipeline Management (DPM), which lets you run OTEL-based transforms directly on the SecOps ingestion path. The mental model: Bindplane-style filtering and routing logic without standing up and operating a Bindplane fleet. You write a transform pipeline, attach it to the relevant GCP collectors, and shape events before they ever land in your tenant. For Cloud Audit Logs, this is the right place to drop the noise that survived your sink filter, and the only clean place to redact fields without dropping the event entirely.

So the design is layered:

1. Sink filter at GCP drops events you have certainty about at the source. This is your primary opportunity to reduce volume, because events dropped at the sink never enter the egress path or the SecOps ingestion plane at all.
2. DPM/OTEL transform catches the residue: events that are too contextual for a static sink filter, or fields that need to be redacted or normalized rather than dropped wholesale.
3. Detection engineering assumes everything that survived (1) and (2) is intentional and queryable.

Anything else you do (Pub/Sub teeing, multi-destination routing, in-line enrichment) outside of SecOps sits adjacent to this spine, not in place of it.

What you never filter

Before talking about what to drop, the bright lines:

• All Admin Activity, every project, every service. This is the detection bedrock.
• All System Event logs. Volume is negligible, and the events catch operations you didn’t initiate.
• All Policy Denied events. Denials are some of the highest-value signal you’ll ingest.
• All Access Transparency events. Every record is Google personnel touching your data. Volume is trivial; signal is absolute.
• All Data Access events from IAM and KMS. These are the closest things to “someone is touching something across my auth surface.” Drop these and you’re blind.
• Break-glass and privileged-role authentication events, regardless of source. If your org_admin role performs an action, you want to know.

If your filter touches any of the above, stop and recalibrate.

What to surgically drop

The win comes from precision, not breadth. The pattern I deploy looks like this: keep all Admin Activity, System Event, Policy Denied, and Access Transparency logs unconditionally, then tune Data Access against a small number of high-volume, low-value, well-understood patterns.

One caveat before the filter itself, because it shapes the most important clause. Dropping bulk storage.objects.get and storage.objects.list events at the sink only works if recon and enumeration detection lives somewhere else in your stack: Security Command Center Enterprise, Wiz, Prisma Cloud, or another CSPM/CDR platform that writes those detections natively and forwards the alerts back into SecOps as findings. If SecOps is your detection plane for that class of attack, keep the events. Same goes for anything else you’re tempted to drop. You’re not deciding whether to detect. You’re deciding where.

A representative aggregated-sink filter (non-exhaustive, but shows the approach), written for the Logs Explorer interface where the sink filter is defined:

Note: Cloud Logging’s filter language doesn’t accept comments. The # lines below are inline for readability. Strip them before pasting into the sink filter.

# Always-on tier - never filter
log_id("cloudaudit.googleapis.com/activity")
OR log_id("cloudaudit.googleapis.com/system_event")
OR log_id("cloudaudit.googleapis.com/policy")
OR log_id("cloudaudit.googleapis.com/access_transparency")

# Data Access tier - keep, except known noise
OR (
  log_id("cloudaudit.googleapis.com/data_access")
  AND NOT (
    # One example of cutting into Data Access, not the only one:
    # Drop bulk GCS reads from CI/CD principals on artifact and state buckets.
    # Only safe if recon/enumeration detection lives in SCCE, Wiz, or similar
    # and the alerts are forwarded into SecOps. Otherwise, keep these events.
    resource.type = "gcs_bucket"
    AND protoPayload.methodName =~ "^storage\.objects\.(get|list)$"
    AND protoPayload.authenticationInfo.principalEmail =~ "^cicd-.*@.*\.iam\.gserviceaccount\.com$"
    AND resource.labels.bucket_name =~ "^(artifacts|build-cache|tfstate)-"
  )
  AND NOT (
    # Drop BigQuery metadata-inventory reads from CSPM tooling.
    # CSPMs enumerate schema via Dataset/Table services, not JobService. The
    # methodName regex matches discovery-only reads (Get/List on datasets and
    # tables) and excludes IAM policy reads, which are higher-signal even from
    # inventory tooling. A cspm-* principal hitting JobService is signal worth
    # keeping rather than filtering.
    # resource.type accepts bigquery_(dataset|project|resource) because metadata
    # calls surface under different types depending on dataset scoping. Verify
    # against a sample in your own tenant.
    resource.type =~ "^bigquery_(resource|project|dataset)$"
    AND protoPayload.methodName =~ "^google\.cloud\.bigquery\.v2\.(Dataset|Table)Service\.(Get|List)(Dataset|Datasets|Table|Tables)$"
    AND protoPayload.authenticationInfo.principalEmail =~ "^cspm-(scanner|inventory)@.*\.iam\.gserviceaccount\.com$"
  )
  AND NOT (
    # Drop GKE control-plane ConfigMap reads from system principals in kube-system
    resource.type = "k8s_cluster"
    AND protoPayload.methodName = "io.k8s.core.v1.configmaps.get"
    AND protoPayload.resourceName =~ "/namespaces/kube-system/"
    AND protoPayload.authenticationInfo.principalEmail =~ "^system:.*$"
  )
)

A few things matter about the shape of this filter.

The default is inclusion, with exclusion as the exception. You’re shipping everything in the audit category by default, then carving out specific, bounded patterns. This is the inverse of how a lot of teams write these filters, and it’s the only correct way: the cost of accidentally dropping a high-value event always outweighs the cost of shipping noise you’ll later refilter.

Each NOT clause is fully bounded by principal, method, and resource. None of them say “drop all GCS reads.” They say “drop GCS reads matching this exact automation pattern, in these exact bucket name patterns.” If a human (or compromised account) starts reading those buckets, the filter doesn’t apply, and the event lands.

The filter is co-maintained with the workload, not a one-time setup-and-forget. When a team onboards a new automation service account that hammers data-lake buckets, the filter gets a new clause. When a project is decommissioned, the corresponding clause comes out. Treat this filter like detection content, because that’s what it is.

The DPM/OTEL escape hatch

Sink filters are static. They evaluate per-record against a Common Expression Language (CEL)-style expression and don’t have access to enrichment, history, or programmable logic. Two situations call for something more:

1. You need to drop a field, not the whole event. A canonical case in regulated estates: BigQuery JobService.InsertJob events ship the full SQL query text inside the protoPayload. The audit event itself is high-value (who ran what, against which dataset, with which job config), but the query body routinely contains PII in WHERE clauses (account numbers, member IDs, etc), inflates event size dramatically, and adds nothing to most detections you’d write on top. You want the event, not the query, most of the time.
2. You need conditional logic that depends on multiple fields beyond what CEL expresses cleanly. Sink filters get unreadable past a certain complexity. OTEL transforms stay legible.

A DPM transform sketch for the BigQuery case:

# OTEL transform attached to the GCP collector path
processors:
  - transform/bq_query_redact:
      log_statements:
        - context: log
          statements:
            # Drop the SQL query body on BigQuery job submissions.
            # Keep the audit event itself: principal, dataset, job metadata.
            - delete_key(
                body["protoPayload"]["metadata"]["jobChange"]["job"]["jobConfig"]["queryConfig"],
                "query"
              ) where body["protoPayload"]["methodName"] == "google.cloud.bigquery.v2.JobService.InsertJob"

One caveat that applies to any OTEL transform against GCP audit logs: the exact path under body[...] depends on how your DPM receiver maps audit log fields into OTLP (OpenTelemetry Protocol). It’s almost always nested under body rather than flattened into dotted attribute keys, but the precise structure varies by collector configuration. Ingest one BigQuery audit event end-to-end, inspect it in the collector preview, then write the transform against the actual shape. (I’d rather flag that than hand you syntax that looks authoritative and breaks on first contact.)

Take the YAML above as a sketch, not gospel - it may well need fixing against your tenant. The part worth getting right isn’t the keys and paths but the choice they encode: what crosses the boundary into your detection plane, in what shape, and what your analysts end up living with.

(You can do equivalent work with Pub/Sub teed into Cribl, GCS bucket sinks driving Bindplane standalone, or any other pre-SIEM pipeline. They all work. They all add operational surface area. Use them when there’s a real reason. Default to native.)

What this discipline buys you

A SecOps tenant fed by a properly-filtered Cloud Audit Log pipeline behaves differently from one fed by a turnkey integration. Detections run against a denser signal-to-noise ratio. Cost is predictable and tied to growth in the estate, not to engineers adding new integrations. Analysts don’t tune out. Storage trends correlate to actual GCP activity, which means you can spot anomalous ingestion as a security signal in itself.

The trap is that none of this happens unless someone treats Cloud Audit Log integration as a detection engineering problem rather than a connector configuration problem. The toggle is the start, not the finish.

The principle

Every log you ingest is a detection bet. You pay in storage, in pipeline cost, and in analyst attention to keep that event close, in exchange for the option to query, alert, or correlate against it later. The events worth that bet are a small, deliberate subset of what GCP will hand you if you let it.

The bet only pays off if it is made with intention.

Related posts

Google Cloud NEXT 2026: Separating Signal from Noise on the Security Releases

A security practitioner's take on what actually landed at NEXT: the agents story is more complicated than Google is letting on, and the real news got buried.

April 27, 2026 · 8 min read