In order to hold an Olympics, a city needs a certain amount of stability and wealth. This can only be guaranteed in North America, Europe and parts of Asia.
Well, the Ancient Greeks managed to hold Olympics before the Industrial revolution. Not quite that stable and wealthy a society, although it was stable and wealthy enough to be a legend to us.
The modern Olympics have very different requirements from ancient Greece. There's a very long list of specific criteria: venues for the modern olympic sports; local transit; hotel rooms; airports; telecommunications; security. The ancient Greeks weren't concerned with any of this.
Again, I'd advise those who are curious to read the publicly available report evaluating the nine original candidate cities. It's an education.