Yeah you really have to pull at that thing. It's strong though, you won't break it. The rubber gets old and turns to stone. Maybe popping it out when the engine is hot would be a good idea.
It will take some muscles but once the pcv is out, make sure the rubber doesn't drop into the head cover. I did that once and it was alot of work to get it out.