It means you should not follow the path of an ascetic, abstaining from all pleasures, but on the other hand also not indulge too much in pleasure. Just live a normal life.
Buddha: “Monks, these two extremes ought not to be practiced by one who has gone forth from the household life. (What are the two?) There is addiction to indulgence of sense-pleasures, which is low, coarse, the way of ordinary people, unworthy, and unprofitable; and there is addiction to self-mortification, which is painful, unworthy, and unprofitable.
Avoiding both these extremes, the Tathagata (the Perfect One) has realized the Middle Path; it gives vision, gives knowledge, and leads to calm, to insight, to enlightenment and to Nibbana (nirvana). And what is that Middle Path realized by the Tathagata…? It is the Noble Eightfold path, and nothing else, namely: right understanding, right thought, right speech, right action, right livelihood, right effort, right mindfulness and right concentration.”