I had the same problem...
it's some kind of scale. Check the under side of those leaves, look along the center rib. Are there flat reddish thingies along the rib?
I had to get down and dirty. Took the whole plant and put it into the tub, gave it a shower. Then, I took Q-tips and rubbing alcohol and cleaned each leaf and the branches.
Then I sprayed the whole tree, branches and dirt included, with a 3-in-1 Fungicide, Insecticide, Mitecide. I bought the stuff at Home Depot, but WalMart also had it.
I've kept up the spraying since then and so far, no return of nasty scale thingies...
Good luck.
BTW, my tree has loads of fruit on it.
Anyone know how to tell when they are ripe? Or doesn't it matter?