The man, who also now lives in the South, was speaking on South Korean broadcaster JTBC, which described him as the manager of a restaurant in China where the women had been working.
He was not identified but said he had carried out the group’s escape with the help of South Korea’s National Intelligence Service.
He also said the women did not know where they were going.
This was confirmed by four women who told the same TV channel they were part of the group.
The women said they only realised what was happening when they arrived at the South Korean embassy in Malaysia.
They also pleaded to see their parents in North Korea again.
At the time of their defection in 2016, South Korea had said it was satisfied that the women had escaped from the North voluntarily.
But North Korea had accused the South of “hideous abduction” and demanded the group be returned.
South Korea’s Unification Ministry spokesman Baik Tae-hyun said: “New claims have been made on how the North Korean workers escaped and arrived in South Korea and whether they were acting on their free will.
“For now, I can only say there’s a need to confirm whether these claims are true.”
He did not say whether the women might be returned to North Korea if it is confirmed they were in the South against their will.
But he admitted the South had never interviewed them after their arrival in 2016.
An official for South Korea’s spy agency did not confirm or deny the JTBC report.
According to the South Korea government, more than 30,000 people have fled from the North to the South since the end of the Korean War in 1953.
North Korea runs more than 100 restaurants in other countries, where the mission is to earn foreign currency and provide a much-needed source of income for the financially-isolated country.
As well as serving North Korean food, restaurant workers often dance in traditional costume and sing North Korean songs.
More from North Korea
Donald Trump hails Kim Jong Un as North Korea prisoners arrive home
Kim’s captives arrive back in US
Time and place for Trump-Kim summit to be announced ‘within three days’
Kim Jong Un strolls along seaside with President Xi Jinping in China
North Korea resets clocks by half an hour to match South
Trump meets Kim Jong Un: Date and location set for historic summit
Reports say that those chosen for restaurant work are from elite Pyongyang families and their patriotism is key.
They often work long hours and have little contact with the world outside the restaurant.