Free Will and Predetermination

I went to the Beyond BT melave malka last night where Rabbi Lazer Brody spoke. It was a nice crowd and he is an excellent speaker (meaning, he really managed to hold my attention, my mind has a tendency to wander during shiurim). He spoke about something that has bothered me for a long time. He didn't clear it up for me, I still have major issues with it. He spoke about Hashem's plan, and how He means for everything to happen the way it does, but along with that, the fact that we each have free will, because while Hashem knows what is going to happen, we don't. I have a very hard time comprehending how these two concepts can not be mutually exclusive. Either everything is predetermined, and therefore we don't really have a choice in our decisions, or we have free will and Hashem doesn't plan out everything. I have heard arguments that they don't have to be mutually exclusive many times, but I have yet to get my brain around it. And what is really bothering me these days is this: If Hashem determines everything that is going to happen, why would He allow us to make decisions that go against His will? Especially because, as a Torah-obsevrant Jew, I know what Hashem wants from me, but there are times when I make decisions that run counter to His will anyway. If Hashem wants the best for me, why would He predetermine, along with giving me the free will to make the decision, that I commit actions that run counter to his mitzvos and in effect, harm myself, at least spiritually, if not in many other ways? I have a hard time with both sides of the argument. I don't want to give up my free will and feel like everything I do was predetermined and that I have no choice in anything. Because to me that is a robotic life and not really worth living. There are no lessons to learn when you don't make your own decisions, though of course Hashem could implant the lessons He wants, I don't limit Him in that way. On the other hand, if we truly have free will, and make our own decisions, that certainly seems to limit Hashem's power to the will of humans to do whatever they want. And if I am to believe that I should follow all of Hashem's commandments as put forth in the Torah, then I don't want Hashem to be limited in any way. Because why should I follow a limited being - that would just make Hashem like us. Maybe a higher form of human, maybe more knowledgable and wiser, but still limited. That bothers me. And again, if everything is predetermined, if Hashem is all-powerful, then why does He allow us to make mistakes, to lead our lives counter to His wishes? Why does He allow me to, over and over again, make decisions that I know are wrong, that I know go against Torah, that I know just serve to defeat myself? Again, maybe this is the lesson I am here to learn, that I need to strengthen myself against these decisions that cause harm to myself, and that is ultimately the plan - that I should grow in this way by making these mistakes. And maybe I need to learn this in order to teach others and to share in the struggle by being able to relate to others who go through the same struggles. And what of those who either grow up and never find Torah, which is supposedly the way Jews should lead their lives, or those who choose, for whatever reason to go "off the derech" and never come back? Is that predetermined to happen? And why? Why would Hashem want people to never find His way? And if it's not predetermined and we have free will to make such decisions, then again, it seems as if Hashem's power is limited in who He can touch. These are tough questions, with no simple answers, other than those that get twisted in knots or are not challenged. I haven't found solutions yet. Anyone care to try?