Can perf numbers be trusted?


  • 0
    A

    Submitted a C++ solution for a problem - 3ms.

    Changed the code a bit, submitted - 6ms.

    Restored the original code, submitted - 6ms

    I wonder if the perf measuring scenarios run the tests several times, taking averages, making sure deviation is small. If the tests run on virtual machines, how dedicated are those to our submissions?


  • 0

    @andhddn First, you posted in the wrong category. This category is strictly for posting interview questions only. I have now moved it to "General Discussions" category.

    Please read the following for more instructions:
    https://discuss.leetcode.com/topic/22/welcome-new-users-please-read-this-before-posting

    As for your question, the time difference between 3ms and 6ms, although 100% difference, but 3ms in CPU time is nothing and could be caused by many reasons. For example, there may be interrupts happening which adds to the overhead in CPU time.


  • 0
    A

    Thanks for moving the topic and for your answer. Today i have another evidence that the perf comparisons can't be trusted or... maybe people making statements can't be trusted.

    1. Submitted the code for integer palindrome testing. Got 75ms results.

    2. Went into discussion. Noticed some code with a claim of 46 ms. Copied the "competitor" code into my Visual Studio project.

    3. Ran comparison on my numbers (basically a loop testing each integer [0...100 000 000[ plus a couple of billion of other specific numbers.

    4. My time is 1.9s, the "competitor" is 8.2 s.

    I wonder what happens with me submitting the code today and previous code submitted ~2 years ago. Is the hardware always the same? Are the tests recalculated periodically? They can't be all recalculated with each submission.


  • 0

    @andhddn No, the hardware does not stay the same. We recently upgraded our machines, that's why you saw the distribution graph disappeared for few weeks and came back recently.


Log in to reply
 

Looks like your connection to LeetCode Discuss was lost, please wait while we try to reconnect.