You are right in saying that Yoga was focused toward physical, mental and spiritual development of a person. But we have to remember that the goal of Physical and Mental development was also to be able to use them for Spiritual Development. In other words a Yogi would tune up his Body and Mind so that he can practice Spirituality. After all the basic teaching is that we are not this Body or Mind. We are spirit souls.
In vedic days, there where a class of persons called Kshatriya who were very willing to fight for sake of Justice. Such persons were trained in Martial Arts but not the whole society. Some where intellectually inclined, some economically whereas some where simply happy with some simple job.
So though it is a good idea that it be introduced and taught in schools, it should not be compulsory for every one.