Drivers...
It is what held back the Linux Zaurus, the N900, and it is also what wrecks a libre situation with android hardware.
If we could get useful drivers for the linux kernel we would have a stopgap for an android device until the kernel the driver was built for gets old and moldy.
With FOSS drivers we can port them to just about anything, but android goes one worse and doesn't even use a fully compatible kernel and it seems that convergence of the android and mainline Linux kernel has been put way off or abandoned.
No drivers and no kernel mean half or at least several components of the hardware in your phone/tablet in most devices is unsupported or you can't even boot a Linux which uses normal sound and video.