Why does gcc -O3 produce wildly different assembly for the same function?

I have a loop in a physics engine which detects collisions like this:

  // now check for collisions
  // we only allow 1 collision per 2 partcles per frame so the
  // one with the lower index will always "collide" first
  for (size_t j = 0; j < m_particles.size(); j++) {
    for (size_t k = j + 1; k < m_particles.size(); k++) {
      m_particles[j].collide(m_particles[k]);
    }
  }

I made a change to use a double buffer for the particles internally. (As an aside; this was done so that I can detect errors and go back 1 time step to correct). The double buffers were implemented with 2 pointers which I swap after this loop. So now we have:

  // now check for collisions
  // we only allow 1 collision per 2 partcles per frame so the
  // one with the lower index will always "collide" first
  for (size_t j = 0; j < m_current_particles->size(); j++) {
    for (size_t k = j + 1; k < m_current_particles->size(); k++) {
      (*m_current_particles)[j].collide((*m_current_particles)[k]);
    }
  }

std::swap(m_current_particles, m_previous_particles);

In both cases, the containers are std::vector.

With the same driver program, the second is over 10x slower. Nothing inside the collide() function has changed, and I can reproduce by creating a build with only the switch to the double buffer and no other changes at all.

Eventually I broke out the profiling tools and sure enough... the assembly for the collide() function is completely different, despite identical source. In either case it there is an explicit callq to a collide subroutine, so it's not inlining the collision function into the loop in the former case.

In the former case:

• All Vector operator overloads on the particles' Vector (my own vector implementation, not a std::vector) members have been translated to inline assembly.
• Makes heavy use of the %xmm registers
• Composed of a bunch of dedicated floating point multiply, add, sqrt, instructions.
• Self contained subroutine, only has conditional jumps back to be beginning as you'd expect for something like this.

In the performance regression case:

• Minimal inlining, context switches to call subroutines for all Vector operations.
• Almost no use of %xmm registers, everything is %rax, %rbx, etc.
• Even inspecting the subroutines, they seem to be not as well optimized (a lot more assembly, not a lot of usage of dedicated mul/sqrt instructions).
• Context switches many times which incurs a huge performance overhead.

Try as I might I cannot convince the compiler to produce something like the original assembly with the new code.

• Tried marking everything inline which is called in collide() (although I hear these days the compiler largely ignores you if you do that).
• For fun, tried manually writing all the math by just accessing members and doing it in the function instead of calling their functions / operators. No dice. Did kick me down to about a 7x performance hit instead of 10x, though.
• Messing about with calling the function in varying ways inside the loop.
• Making an auto& local_particles = m_current_particles local in case the compiler was upset about the scope of the pointer.
• Making a copy of the array locally and calling with that, making it look the exact same as the original loop, then copying the array back. No luck, and this wouldn't scale even it it resolved.
• Pleading with God.

Is this just a case of the compiler's insane brain lining up in just the right way in the former case, or has my calling convention somehow broken an invariant I've held to be true between these two sources?

Here's the relevant parts of the call graph:

Former case (everything's been inlined):

Regression (explicit calls are obvious):

Needed to be collapsed to fit reasonably, the calls below collide() are the operator- overload for Vector and the Vector constructor itself to create the temporary object.

Edit: With -O2 on the former case, it inlines the the expensive vector subtraction operator but not the ctor call for the temporary. Runs in about 1.3x vs -O3. -O2 and -O3 in the regression case seem essentially the same.