It's as if you were trying to put together a 200-piece jigsaw puzzle and you had 100 extra pieces from several other puzzles on the table as well. Before you can start putting the puzzle together, you have to separate the pieces that fit your puzzle and set the others aside.
By contrast, many of my colleagues at Goddard's creative writing program take an opposite approach. They might start, as I do, with the germ of an idea, an image or a snippet of conversation. But from there they are more likely to plod along, figuring out, "what happened next," and so on and so forth until the story reaches its culmination. When they're done with a first draft, they have a summary of a story which they then need to flesh out with scenes and description in future drafts. In our puzzle analogy, they would start by pulling out all the edge pieces first and putting them together.
There's no right or wrong way to write. What's important is knowing your process. Right brain or left? Creative or logical? Plain or peanut? As long as you're writing, it's all good.