At work we have several PHPUnit tests that randomly fail. These ghost failures cause a lot of pain. One run the test fails, the next it passes. These tests are slow poison, as they degrade trust in the system. The code was inspected, but nothing could be found wrong. For some reason, on random test runs, the arrays being returned would have their elements in a different order, but no one knew why.

The order of the arrays clearly mattered in our tests, so what was causing them to change that we didn’t expect?

Short version:

In the tests we were using PHPUnit’s assert functionality to test that two arrays were equal. If the ordering between the two arrays is off, even if they contain the same key value pairs, PHPUnit will fail the test. Comparing arrays in this way normally works, because PHP usually guarantees array order; however, under some circumstances order is not guaranteed. This was the cause of the random failures. To fix this, we just need to be smarter about how we compare arrays.

Long version:

In investigating one of the tests that was failing randomly, I saw that it was performing an assertEquals on two arrays. Sometimes this assert would fail because the ordering in one of the arrays had changed for that test run.

The question that is relevant here is this: does PHP guarantee the order of arrays?

The answer: it depends.

PHP arrays are implemented under the hood as an ordered map – what is essentially a Linked Hash Map in Java. Insertion order is guaranteed. However, PHP breaks this contract for some array functions, such as array_slice, which does not guarantee order unless a flag is set.

In the case of the test I investigated, the final array was being generated by an array_slice that was buried in the code. One potential fix is to change the code under test so it does not use array_slice or so the flag to maintain order is set. The other fix is to make the tests smarter.

I opted to go with option two, making the tests smarter. This can be done by using compare functions such as array_diff or array_diff_assoc and then asserting on the output of the compare function instead of asserting for equality between two arrays. Another option is to sort the output array, so it always has the same order. Your creativity is the limit on how you solve this.

What does this episode indicate? That it is worth understanding how the language you’re using operates. While PHP has more warts than most, all languages have idiosyncratic behavior that it is worth being aware of. The next time you see perplexing behavior, ask yourself why it’s happening and investigate. The reasoning why the language is behaving as it is might surprise you.