"That's Life" - Is the most polite way I can say it.
Neighorhoods, along with civilizations, empires, countries, even the stars, are born, go through a period of maturation, decline, and eventually die. The only constant in life is change.
People like to bring up these false arguments against gentrification, as if it's better that a neighborhood remained "authentic" and mired in poverty, rather than "outsiders" (99% of the time this is a code word for "whites", "gringos", "gabachos", "crackers", "honkies", whatever the current term used in the Third World Studies Department at UC Berkeley to disparage those of European background) come in and the neighborhood shows signs of improvement.
People also seem to forget that neighborhoods in the New York City area are in a constant state of flux. What about the neighborhood known today as Harlem? Of course we all know the Indians were there first. If there's anyone who has a legitimate "beef" with gentrification, it starts and ends with the Indians who were indeed "here first"! According to Wikipedia, Harlem was named by the Dutch after the city in Holland known as Haarlem (two A's), back in the 1600's. The Dutch were eventually displaced by the English. Fast forward to the early 1900's, and Wikipedia continues noting the arrival of East European Jews, which reached a peak of 150,000 in 1917. Jews were eventually displaced with the arrival of Blacks in large numbers from the South. And, in the past 10-20 years, Harlem is still going through changes, with whites returning to buy brownstones/buildings, even our "first black president" Bill Clinton has an office in Harlem. Starbuck's, Ben and Jerry's, Magic Johnson Cinemas, Gap, etc. etc. all have stores in the Harlem neighborhood now. Walk through Harlem and you'll also see signs of other recent new comers from Mexico, Central America, west Africans, etc. etc.
What about the Washington Heights neighborhood? Present day it's an area of mainly Dominican immigrants, but Alan Greenspan (former chairman of the Federal Reserve), and Ruth Westheimer ("Dr. Ruth" the "sex" doctor), lived there in their youth.
The point here is that neighborhoods change. What would your driver propose should be done? Have government step in and force everyone to live in areas designated by social class? By race? By income? Would he expect government to curtail our freedoms in order to prevent "gringos" from moving to Jackson Heights and "raising the cost of everything"? Remember also, that things go both ways. Just as gentrification is a real phenomena, neighborhoods also go through periods of decline. "Slumification", "white flight", etc. etc., those are real phenomena just the same. Oh, I forgot, I'm not supposed to bring up that, I should have paid attention to those Marxist classes I was forced to take as electives in college!
Also, for the most part, my personal opinion is that "folks from Manhattan" (polite euphemism for "whites") are for the most part buying co-ops in Jackson Heights, rather than renting. People buying co-ops generally have little impact on the rental market for apartments, these are 2 different market segments. Additionally, "folks from Manhattan" is a pretty broad, general term. Most of the "folks from Manhattan", or "whites" if we want to be blunt, in my co-op building are retirees, immigrants from eastern Europe, widowers, people of fairly modest means. In fact, 3 of the "folks from Manhattan", including myself, are married to persons of non-European origins!
Anyways, if your driver is being "muscled" by his landlord doing something illegal to get him out of his apartment, there are plenty of organizations and city agencies able to intercede on his behalf, if he's willing to do the research. But blaming the "gringos" isn't going to help him or anyone else.
The other alternative is for the United States to set up a system like North Korea, where there is no freedom of movement, or any type of freedom, but at least everyone is pretty much equal. But I don't think even your driver is willing to drive down that road!