I got a CD player this year and have really enjoyed being able to throw on an album sans phone when hanging out with friends or cooking. If you're just getting into it, start with a shittier CD player and see if you actually use it before investing in a nice one. My CD player also has a bluetooth feature for the moments I feel the urge to play something off of streaming.
In terms of where to buy, thrift thrift thrift. CDs are like $1 at Goodwill and they usually have decent selection (got my fav Cranberries CD from a Goodwill). Also!!! used bookstores are a great place to look for CDs. Also it's fun to grab CDs by artists you recognize but maybe albums you're unfamiliar with- great way to find new stuff