What went wrong here?

Well it sounds as if you are more than simple fwb and you have to figure out what your relationship is. On the one hand you are spending alot of time together and you also seem to be very sensitive in regard to your partner's opinion of you. These aspects are more like a relationship. Perhaps you should consider either being simply fwb's or try to take the relationship to another level. Good luck!