Evaluation

Evaluation

Run eval.py with the relevant arguments to produce .h5 files which contain the information needed to evaluate the performance of the training.

python eval.py --config <model_save_dir>/classifier.yaml --gpus 0 --sample ttbar

By default, the evaluation script will look inside <model_save_dir>/ckpts for the training checkpoint with the lowest validation loss, and will use this checkpoint to evaluate. If you want to evaluate at a specific epoch, you can use the --ckpt_path argument.

You may also want to adjust the --batch_size and --num_workers if fewer computational resources are available as compared with when training.

Only one GPU is supported at the evaluation stage.

Make sure you only use a single GPU (or just the CPU).

Plotting

In gnn_tagger/evaluation/, you can modify the plotting configs in configs/ and make plots by running

python make_plots.py --config configs/plot_config.yaml 

By default, plots are written to a new timestamped directory under plots/ but you can specify the output dirname with --dir. Run with --help for more information.

Choosing flavour fractions

Inside the plot_config.yaml, you can specify $f_c$ and $f_b$ for each model. You can also use set the fc to auto to attempt to automatically calculate the best value. This can have significant effects on the results, as it allows for different values of fc to be used for the ttbar and zprime samples.

Supported Folder Structures

When loading models with source: umami, you can provide a dataset_version. By default, the code looks for a hybrids/ directory inside the dataset_version directory, and tries to load umami hybrid samples from it. You can also load files directly from TDD dumps. If the hybrids/ directory is not present, the code will look for subdirs containing ttbar|410470 or zprime|800030, and then loads the h5 files found in those subdirs.

In the example below, dataset_version: dataset_version_1 would load the hybrid samples, and dataset_version: dataset_version_2/source_data would load the TDD h5 files.

.
├─ save_dir/
│  └─ dataset_version_1/
│     └─ hybrids/
│        ├─ MC16d-inclusive_testing_ttbar_PFlow.h5
│        └─ MC16d-inclusive_testing_zprime_PFlow.h5
│  └─ dataset_version_2/
│     └─ source_data/
│        ├─ mc16_13TeV.410470.../
│        └─ mc16_13TeV.800030.../

Tracks

You also plot track classification ROC curves as below.

python make_plots.py -c configs/plot_config_tracks.yaml --num_jets 1e5

You don't need to load as many jets to get good track level stats, so using --num_jets speeds things up a bit.