Detect and Alert on Suspicious Container Behavior at Runtime with Falco

What You'll Build

Deploy Falco as a DaemonSet on Kubernetes using Helm, write three custom rules that catch realistic attack patterns (shell injection, package manager use, and sensitive filesystem writes), and route live alerts to Slack through Falco Sidekick.

Prerequisites

• Kubernetes 1.27+ cluster with kubectl configured and cluster-admin access
• Helm 3.12+
• Nodes running kernel 5.8+ (verify with kubectl debug node/<node-name> -it --image=ubuntu -- uname -r; required for the modern eBPF driver)
• A Slack incoming webhook URL from api.slack.com/apps

This tutorial targets Falco 0.37.x via its Helm chart. On managed Kubernetes (EKS, GKE, AKS), the modern eBPF driver is the right call since those environments block kernel module loading by default.

1. Add the Helm Repository

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

Dump the defaults for reference before overriding anything:

helm show values falcosecurity/falco > falco-defaults.yaml

2. Write Your Values File

Create falco-values.yaml. This sets the driver, enables JSON output, injects custom rules via a ConfigMap, and wires Sidekick to Slack.

driver:
  kind: modern_ebpf   # No kernel module; requires kernel >= 5.8

falco:
  json_output: true
  json_include_output_property: true

customRules:
  custom-attack-rules.yaml: |-
    - list: pkg_mgmt_binaries
      items: [apt, apt-get, aptitude, yum, dnf, pip, pip3, npm, gem, apk]

    - rule: Package Manager in Container
      desc: A package manager ran inside a live container. Common post-exploitation behavior.
      condition: spawned_process and container and proc.name in (pkg_mgmt_binaries)
      output: >-
        Package manager in container
        (user=%user.name container=%container.name
        image=%container.image.repository cmd=%proc.cmdline)
      priority: ERROR
      tags: [container, process, mitre_persistence]

    - rule: Shell Spawned in Container
      desc: A shell started with a non-init PID inside a container, suggesting an exec-based intrusion.
      condition: >
        spawned_process and container and
        proc.name in (bash, sh, dash, zsh, fish, ksh) and
        proc.vpid != 1
      output: >-
        Shell spawned in container
        (user=%user.name shell=%proc.name parent=%proc.pname
        container=%container.name image=%container.image.repository
        cmdline=%proc.cmdline)
      priority: WARNING
      tags: [container, shell, mitre_execution]

    - rule: Write to Sensitive Path in Container
      desc: A process opened a file for writing under /etc or system binary directories.
      condition: >
        open_write and container and
        (fd.name startswith /etc/ or
         fd.name startswith /bin/ or
         fd.name startswith /usr/bin/) and
        not proc.name in (dpkg, rpm, apk)
      output: >-
        Sensitive write in container
        (user=%user.name file=%fd.name proc=%proc.name
        container=%container.name image=%container.image.repository)
      priority: WARNING
      tags: [container, filesystem, mitre_defense_evasion]

falcosidekick:
  enabled: true
  config:
    slack:
      webhookurl: ""          # Injected at install time; do not hardcode here
      minimumpriority: "warning"

A few things worth understanding in these rules:

container is a built-in Falco macro equivalent to container.id != host. spawned_process matches execve syscall exits. open_write matches file-open syscalls with write intent. These ship with Falco's default ruleset; you don't define them yourself.

The proc.vpid != 1 filter on the shell rule matters more than it looks. Many legitimate containers run a shell as their entrypoint (PID 1 in the container's namespace). Filtering that out eliminates a large category of false positives without needing an ever-growing image exclusion list. When someone exec's into that same container later, the new shell gets a different virtual PID and the rule fires correctly.

The sensitive-write rule excludes package managers by name because some base images run them during container init. Extend that list as you discover noisy procs in your environment.

3. Deploy Falco

Pass the Slack webhook URL via --set so it never touches disk in plaintext:

helm install falco falcosecurity/falco \
  --namespace falco \
  --create-namespace \
  -f falco-values.yaml \
  --set falcosidekick.config.slack.webhookurl="https://hooks.slack.com/services/T.../B.../..."

Wait for the DaemonSet to roll out:

kubectl rollout status daemonset/falco -n falco --timeout=120s
kubectl get pods -n falco

Expect one falco-* pod per node and a single falco-falcosidekick-* pod from the Deployment.

Advertisement

AirDroid Personal Best mobile device management suite — only $3.99/mo →

4. Trigger Detections

Fire the shell rule by exec-ing into any running container:

POD=$(kubectl get pods -A --field-selector=status.phase=Running \
  -o jsonpath='{.items[0].metadata.name}')
NS=$(kubectl get pods -A --field-selector=status.phase=Running \
  -o jsonpath='{.items[0].metadata.namespace}')

kubectl exec -n "$NS" "$POD" -- sh -c 'id'

Fire the package manager rule with a throwaway pod:

kubectl run falco-test --image=ubuntu:22.04 --restart=Never -- sleep 300
kubectl exec falco-test -- apt-get update -y
kubectl delete pod falco-test

Verify It Works

Check Falco pod logs for JSON-formatted alerts:

kubectl logs -n falco -l app.kubernetes.io/name=falco --tail=30 | grep "Shell spawned"

Expected output shape:

{"hostname":"node-1","output":"Shell spawned in container (user=root shell=sh ...)","priority":"Warning","rule":"Shell Spawned in Container","time":"2024-...T...Z"}

Then confirm Sidekick forwarded it:

kubectl logs -n falco -l app.kubernetes.io/name=falcosidekick --tail=20

A successful delivery shows [INFO] Slack - Post OK (200). If that line is there, the alert is in your Slack channel.

Troubleshooting

Falco pods crash on startup with an eBPF load error. The modern eBPF driver needs kernel 5.8+. Drop to driver.kind: ebpf (the legacy probe, compatible with 4.14+) or driver.kind: kmod if your nodes allow module loading. Run helm upgrade falco falcosecurity/falco -n falco -f falco-values.yaml --set driver.kind=ebpf to switch without reinstalling.

Pods are Running but no alerts appear in logs. Rule syntax errors or invalid field references cause Falco to skip affected rules silently in some versions. Check for errors at startup:

kubectl logs -n falco <falco-pod-name> | grep -iE "invalid|error" | head -20

Also confirm the exec'd pod isn't running a shell as its entrypoint (vpid=1), which the shell rule intentionally excludes.

Alerts in Falco logs but Slack gets nothing. Sidekick logs are specific about the failure. A 401/403 means the webhook URL is wrong or the Slack app lost its permission. A timeout or connection refused means Sidekick can't reach hooks.slack.com, which points to an egress network policy or firewall blocking outbound HTTPS from the falco namespace.

Alert volume is too high to be useful. Use Falco's exceptions key (available since 0.28) instead of piling not clauses onto the condition. An exception block lets you exclude specific images, namespaces, or users surgically without duplicating the rule. See the Falco exceptions docs before your condition grows past three lines.

Next Steps

• Falco Sidekick UI: Enable the bundled dashboard with falcosidekick.webui.enabled: true for a live alert feed and event timeline. Worth running during initial rule tuning to see alert volume before it hits Slack.
• More output destinations: Sidekick supports PagerDuty, Datadog, Elasticsearch, and around 50 other sinks with the same config pattern. Adding a second output is another key block under falcosidekick.config.
• Production hardening: Pin the Helm chart version with --version, store the webhook URL in a Kubernetes Secret referenced by Sidekick's config (the chart supports existingSecret), and set resource requests/limits on the Sidekick deployment so alert routing survives node pressure.
• Admission control pairing: Falco detects anomalies at runtime; it doesn't stop the workload. Pair it with OPA Gatekeeper or Kyverno to reject known-bad workload configurations at deploy time, so Falco handles the unexpected rather than the preventable.