Filtering

What filtering is

A traced run sees everything your process and its children touch — not just your dataset and your model, but the Python interpreter reading its own standard library, pip reading site-packages, CUDA reading driver files, your logger writing to /tmp, and your experiment tracker scribbling into wandb/. Most of that is noise: it isn't part of your pipeline's real input→output story, and if roar recorded all of it every DAG would be buried under hundreds of incidental reads.

Filtering is how roar keeps the recorded lineage to the files that actually matter. It drops well-known noise before it becomes an artifact, so roar dag, roar show, and a registered lineage stay readable and reproducible.

Filtering shapes the record, not the run. Filters only affect what roar records as inputs and outputs. Your command runs exactly as it always would — the same files are read and written on disk. A filtered file is one roar chose not to track, not one it hid from your program.

There are two layers, and they answer different questions:

Built-in noise filters

These are category filters roar applies out of the box. Each is a boolean in the [filters] table of .roar/config.toml, and all default to true.

Turn one off when you genuinely need that category in the record — for example, if an intermediate you care about is written under /tmp:

roar config set filters.ignore_tmp_files false

A read is kept even when its category is filtered, if nothing else explains it. A /tmp file your run only reads (never wrote) is still surfaced — flagged as an unsourced input, because it won't exist when the pipeline is reproduced elsewhere. Filtering suppresses noise, not unexplained dependencies. The safe fix is to not stage real inputs in /tmp.

Project path filters

Built-in filters cover categories that apply to everyone. Your project also has its own noise — an experiment tracker's wandb/ directory, a logs/ folder, a scratch workspace. That's what roar filter is for. It manages the ignore_paths list in a .roarconfig file at your project root:

roar filter add wandb/ logs/ .cache/wandb/   # add patterns
roar filter list                             # show current patterns
roar filter remove wandb/                     # drop a pattern

roar filter list prints the file it reads from and the patterns in it:

.roarconfig (/home/you/project/.roarconfig):
  wandb/

Pattern matching is deliberately simple, matched against the file paths the tracer observes:

• An absolute pattern (starting with /) matches as a prefix: /data/scratch/ ignores anything whose path begins with it.
• A relative pattern matches as a substring anywhere in the path: wandb/ ignores any path containing wandb/.
• A trailing / means "this directory and everything under it."

.roarconfig is meant to be committed. It lives at the project root, separate from the local, git-ignored .roar/ state directory. Checking it in means everyone tracing the project — and every CI run — shares the same project-specific filters, so lineage looks the same for the whole team.

Seeing what was filtered

By default roar run reports filter activity as a one-line count so nothing is silently dropped:

hint: 10 files under /tmp ignored (filters.ignore_tmp_files).
hint:   to track them: roar config set filters.ignore_tmp_files false

To see the individual paths that were filtered, raise the output verbosity:

roar config set output.verbosity debug   # also lists every filtered file (can be large)

output.verbosity accepts quiet, normal (the default — status plus filter counts), verbose (also lists tracked reads/writes), and debug (also lists filtered files).

Strict mode: cleaning up /tmp writes

By default roar ignores /tmp writes (filters.ignore_tmp_files) but leaves them on disk. Strict mode goes further: it deletes the /tmp files a run wrote once the run finishes, so scratch output doesn't linger between runs and can't be mistaken for a real input later.

roar config set cleanup.delete_tmp_writes true   # off by default

Strict mode (cleanup.delete_tmp_writes = true) overrides the passive ignore_tmp_files behavior — instead of merely not tracking /tmp writes, roar actively removes them. It only touches files the run itself wrote under /tmp; it never deletes anything your run only read. Leave it off unless you specifically want that scratch cleanup; it's a convenience for keeping a shared machine tidy, not a correctness requirement.

Filtering vs. secret redaction

Path and category filtering decide which files enter your lineage. A separate mechanism, omit filtering, decides which values are scrubbed from a lineage before it's published to GLaaS — API keys in environment variables, credentials embedded in a git remote URL, and explicitly listed secret values. That runs at roar register / roar put time and is controlled by the registration.omit.* settings, not [filters]. See Authentication and Scopes for how registration handles secrets.

What to reach for

• A whole category of files is cluttering every run → toggle a built-in filter (filters.*).
• One directory in this project is noise → add a path filter (roar filter add), and commit .roarconfig so the team shares it.
• An expected input or output is missing from the DAG → check the filter counts in the roar run output or run with output.verbosity=debug; a built-in filter (often ignore_tmp_files or ignore_package_reads) is the usual cause. See Troubleshooting.