I think after a while, people just start to adapt and as many started using the bigger 4.7-inch and 6.1-inch screen sizes, they just got used to it.
Perhaps, if Apple released the Mini much earlier (and at a cheaper price point), or released it as the 2nd generation SE, it would have been much more successful.