Researchers suggest DNA from this group makes up between 2% and 19% of modern West Africans' genetic ancestry.
They believe the interbreeding occurred about 43,000 years ago.
Scientists found links to the Mende people of Sierra Leone, Yoruba as well as Esan people in Nigeria, plus other groups in western areas of The Gambia.
The new study was published in Science Advances this week.
It suggests that ancestors of modern West Africans interbred with a yet-undiscovered species of archaic human, similar to how ancient Europeans mated with Neanderthals, and Oceanic populations with Denisovans.
The research sheds more light on how archaic hominins added to the genetic variation of present-day Africans, which has been poorly understood even though it is the most genetically diverse continent.
Hundreds of thousands of years ago there were several different groups of humans including modern humans, Neanderthals and Denisovans.
The newly-discovered "ghost population" of ancient human species seems likely to have diverged from these groups.
Sriram Sankararaman - the computational biologist who led the research at the University of California in Los Angeles - told BBC Newsday he believed more such groups would be found in the future.
His team looked at the genetic make-up of West Africans and found that some of their DNA came from an ancient unexplained source.
"As we get more data from diverse populations - and better quality data - our ability to sift through that data and excavate these ghost populations is going to get better," Mr Sankararaman said.