From 6a5028e536a26c6a4556f4f1ed76550eb281d427 Mon Sep 17 00:00:00 2001 From: Tanner Robison Date: Mon, 15 Jun 2026 15:30:03 -0700 Subject: neurobench paper notes Just took some notes when reading through the neurobench paper --- neurobench_notes/system track.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 neurobench_notes/system track.md (limited to 'neurobench_notes/system track.md') diff --git a/neurobench_notes/system track.md b/neurobench_notes/system track.md new file mode 100644 index 0000000..5aa7dad --- /dev/null +++ b/neurobench_notes/system track.md @@ -0,0 +1,16 @@ +## metrics +**Correctness:** must be measured to verify the validity of the solution. No thresholds are imposed so the benchmark leaderboard must be analysed to evaluate correctness - efficiency trade offs of solutions. + +**Timing:** measurements can be either sample throughput or execution time depending on the task. + +**Efficiency:** blah blah blah + +In timing and efficiency measurements both pre and post-processing must be taken into account. + +## Benchmarks +**Acoustic scene classification:** Challenges systems to sort audio into predefined categories based on the envinronmental audio context. This can allow embedded devices to adjust sound equalization profiles, target microphone denoising, and support active noise cancellation. + +Also challenges systems to fulfill technical requirements like always-on and real time operation. timing results report on device average execution time per sample. Power should be reported under idle and active contexts. Idle power measures the system prepared for inference with the model loaded, and active power measures the system running pre-processing or inference. + +**QUBO:** Quadratic unconstrained binary optimization (QUBO). basically its a series of yes or no decisions to find the best outcome. Tests the SUT using data that creates graphs based on 3 parameters, size, density, and random seeds. It runs 5 different versions of each to verify the results. After a specific amount of time neurobench cuts off the test and measures energy consumption and compares the solution to the best known solution. + -- cgit v1.2.3