You just need to be more structured with your training since your time is limited. I wouldn't worry too much about what anyone else thinks, the only thing that really matters is that you're enjoying riding/racing your bike. There are a lot of pick-up cross practices (at least in my area) once the fall comes, you should look for a group to get together with and iron out your 'cross skills.

Steve