this post was submitted on 08 Jul 2023
16 points (100.0% liked)

Programming

17524 readers
489 users here now

Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!

Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.

Hope you enjoy the instance!

Rules

Rules

  • Follow the programming.dev instance rules
  • Keep content related to programming in some way
  • If you're posting long videos try to add in some form of tldr for those who don't want to watch videos

Wormhole

Follow the wormhole through a path of communities [email protected]

founded 1 year ago
MODERATORS
snowe
Ategon
[email protected]
16
How to track automated "performance"-type tests over time? (lemmy.world)
submitted 1 year ago by [email protected] to c/programming
7 comments fedilink hide all child comments
 

I'm pretty familiar with automated tests where you're comparing a received value to an expected value (e.g. basically all unit/integration tests)

in a CI/CD workflow, you handle test failures by failing the whole pipeline, and then that commit/PR/etc has a pipeline that failed next to it.

However, what if I have some kind of "performance" measure I want to track, instead? Something that isn't pass/fail, but rather a set of experimental results over time? (e.g. speed of responses from an API, wins/draw/loss rates on chess bot, confusion matrix scores for a classifier, etc.) Is there a tool that can show that kind of "automated experiment" results in order by git commit, pull request, etc?

I thought about sending the data to some kind of data store with a Grafana front-end, but I was hoping there might be some less "diy" method for creating such a display.

you are viewing a single comment's thread
view the rest of the comments
[–] [email protected] 4 points 1 year ago

I'm pretty interested in this too. I've thought about it in the past, and I think get stuck where you're asking (the post processing and visualizing bit).

I'd thought of having GitHub actions for the measurement, stashing the results as artifacts, then having another workflow that processes the results. Obviously pretty DIY so I'm curious if others have solutions.