I can relate to your situation because I'm a musician and when I started doing office work I always thought of it as my "fall-back" position, never my dream "career"! (My mum told me that if I was going to major in music in college, fine, but at least minor in something practical, so I took professional writing. I'm glad I did!)
However, as the years have gone on I've discovered I enjoy admin/office work and yes, one could definitely make a career of it (I've often thought about branching into desktop publishing and marketing for example).
I also realize, though, how putting so much energy into your day job can leave little for pursuing your passion. But don't give up! You can probably find ways to still fit singing into your life. I think trying to make a living in music would be a bit too stressful for me now (and I DON'T like to teach--that is one reason I turned to office work!) However, if money became no object, I would certainly take more time for musical pursuits!