"Doing nothing is not an option," Mattis said following a conference of defense ministers in Rome that discussed the issue without resolving it. Some governments have expressed little interest in having such militants returned.
"We're gathering up hundreds...of detainees," Mattis told reporters traveling with him. "The important thing is that the countries of origin keep responsibility for them. How they carry out that responsibility, there's a dozen diplomatic, legal or whatever ways. But the bottom line is, we don't want them going back on the street."
The U.S.-backed Syrian Democratic Forces are currently holding thousands of IS detainees, including hundreds of foreign fighters from a number of nations. Last week, they announced they'd captured two notorious British members of an Islamic State cell who were commonly dubbed "The Beatles" and were known for beheading hostages.
Asked specifically about Britain's move to revoke the citizenship of the two fighters, Mattis repeated his admonishment that countries "bear some sense of responsibility" for their citizens.
"The most important thing is we figure out how we're going to deal with this," Mattis said.
U.S. military officials have confirmed that El Shafee Elsheikh and Alexanda Amon Kotey, who grew up in London, were captured in early January in eastern Syria.
U.S. officials have interrogated the men, who were part of the IS cell that captured, tortured and beheaded more than two dozen hostages, including American journalists James Foley and Steven Sotloff, and American aid worker Peter Kassig.
Mattis was asked if the U.S. would take any detainees or if the Guantanamo Bay, Cuba, detention facility was a possible option. He said he wasn't willing to talk about that.
The priority now, Mattis said, is to "define the problem and then we'll get the solution." He said there's a need for an accurate count of detainees and where they are from. He said there is not a "one-way forward" for all the detainees right now - just the sentiment they must be taken off the battlefield.
There are no known Americans among the detainees. But a U.S. citizen who surrendered to U.S.-backed forces in Syria is being held in Iraq amid a protracted legal fight over his fate.
Most of the foreign fighters are from the region, including Iraq and Saudi Arabia.
Hundreds fought alongside IS in recent years as it seized large parts of Syria, raising concerns they'll commit terrorism at home if they return.
Mattis met in Rome with defense ministers from about a dozen other nations participating in the fight against the Islamic State group in Iraq and Syria. The meeting coincided with a coalition gathering of diplomatic leaders in Kuwait, including U.S. Secretary of State Rex Tillerson.
Copyright 2018 The Associated Press. All rights reserved. This material may not be published, broadcast, rewritten or redistributed.