laravel order query on relationship

Hi i am trying to get a user and there images, ordered by the primary column which is either 1 or 0

        $user = User::with('image')->whereUsername($username)->whereHas('image', function ($query) {
            $query->orderBy('primary', 'desc');
        })->firstOrFail();

however this query return the user and there images of which there is three of them but they are always in the same order, which is not the descending order and appears to be in ID order, am i missing something here?

laravel php laravel 5 laravel 4 laravel with laravel tutorial