Because it's not caching the metadata as you said, it has to read each file's ID tags and read the artist, title, track, and other information. When you got a ton of files over a network share, it would take a very long time to finish. And because it's not cached/saved, it starts over everytime you restart cog.
So basically, anyone who stores their media on a network share (like 99% of the people) cannot use cog.
Till it's fixed I guess I'll have to use ITunes which also has it's own share of problems sigh.
Maybe in the future, cog could use the same playlist format as foobar (I think it's .pls format), then I could just copy my existing foobar playlist to Cog so Cog doesn't need to rescan the metadata.