The rare visit comes ahead of the potential summit between the leaders of the two countries next month.
The White House has said the on-off meeting on 12 June in Singapore is “expected” to take place still, despite some jitters.
US officials have said early meetings preparing for the America-North Korea summit have been positive.
Mr Pompeo tweeted a few hours after Mr Kim landed in the city: “Good working dinner with Kim Yong Chol in New York tonight. Steak, corn, and cheese on the menu.”
Kim Yong Chol is the highest-level representative from North Korea to set foot in the United States since 2000.
The official, who will meet Mr Pompeo again on Thursday, arrived at New York’s John F Kennedy airport on a flight from Beijing.
He was later seen entering a hotel in midtown Manhattan.
“The US delegation led by ambassador Sung Kim met with North Korean officials today as well and their talks will
continue,” White House press secretary Sarah Sanders said.
“So far the readouts from these meetings have been positive and we’ll continue to move forward in them.”
“We’re going to continue to shoot for the June 12th and expect to do that,” she added, referring to the original date scheduled for the summit.
Mr Trump last week called off the summit after North Korea expressed anger at comments by US officials.
But the president later said he was reconsidering his position and US, North Korean and South Korean officials have gone ahead with summit preparations anyway.
More from North Korea
North Korean official flying to NYC in bid to save Trump-Kim summit
Trump’s summit cancellation with North Korea could help give peace a chance
Donald Trump: June talks with North Korea could be back on after ‘productive talks’
Why is the North Korea summit cancelled? Trump just couldn’t get into Kim Jong Un’s mind
Donald Trump cancels summit with North Korean leader Kim Jong Un
US stocks fall after Trump cancels N Korea summit
Meanwhile, the outgoing head of US pacific command, who is Mr Trump’s pick to be the ambassador to South Korea, said on Wednesday that North Korea remained America’s most imminent threat.
“North Korea remains our most imminent threat. And a nuclear-capable North Korea with missiles that can reach the
United States is unacceptable,” Admiral Harry Harris said in a change-of-command speech.