Check to see what file:added says against the date/time of the file in question. If it's close to created, modified, or present.

I'm going to guess that file:added is close to present, and close together for all unsorted files?

I am going to raise one point, though. I added an old file to the card and the images app inserted it at the very top (it was old). Moving a file off the device and returning it also inserts it in the timeline. So I'm guessing it works. Since tracker indexes both images and media, it's probably not a tracker issue?