Doing your duty is like planting a seed. It may sprout, grow, give flowers and fruits. But if you keep digging up and checking whether the roots are growing and the shoots are growing, then will it grow? It can never grow.
That is why it is said, don't worry about the fruits right now. It will give results in its own time.
Right now, all that you should do is plant the seed, water it and wait patiently till it grows, flowers and bears fruits.
If you plant a mango seed, will you get mangoes immediately? You will only get it in due course of time. So the Bhagavad Gita says that, don’t look for the rewards, just do your duty right now.