Sweater Re-fashion

I've been wanting to try something like this for a very long time. Combining knitting with an existing sweater. I started out with a well-worn, pale lavender, pullover wool sweater. First I cut the sweater up the front, not in the center, but a bit to one side for an asymmetrical look. Next I used some of my early, lumpy, handspun yarn; spun chunky and uneven from a lovely handpainted BFL roving. It didn't take a whole lot of yarn to knit two narrow bands for the fronts, two chunky cuffs, and a ribbed neck. Using the zig-zag stitch on my trusty Kenmore, I sewed all the hand-knit pieces into place. I found these tasty wooden buttons in my button box. I cut narrow strips of cotton t-shirt fabric for the button loops and sewed those on. I'm very happy with how it turned out! I like the asymmetrical look. However, when the front is unbuttoned, the flap hangs down in a strange-looking way. Not sure how to do it differently. Perhaps with a rounded neckline instead of a v-neck? I will definitely be exploring more along these lines.