Title | | » | Summarize/graph pass1 iteration count for tests |
Assigned To | | » | Kaipo Tamura |
Notify | | » | Rita Chupalov;Brendan MacLean |
Type | | » | Todo |
Area | | » | TestResults |
Priority | | » | 3 |
"Also, I am thinking it would probably be interesting to be able to summarize the Pass1 iteration count for any test, maybe even graph it as a histogram when we have enough data. The minimum number of iterations to prove no leaking is 8, and there will be many tests that achieve that. But, I find when I get a leak notification about a test like today's ConsoleMultiReplicateImportTest where it ran to 25 iterations and looks like something I can't exclude from being a real leak, I wonder how often it is running more than 8 iterations to prove this test is not leaking. i.e. How much of an anomaly is today?
It would be really nice to add more information on this to your new test "Leaks" page option.
I am thinking more historically. Give me distribution information about how many iterations it usually takes to decide this test doesn't have a leak. It may get tested as many as 15 times a day and we are only seeing it break through to being reported as a leak once or twice a month. How many iterations did it need the other 15 * 30 days (i.e. 450). That is enough times to have a histogram ranging from 8 to 25, since 8 is the minimum and 25 gets reported as a leak."