I have had many of the same experiences, but with a twist. I loved history! It was my favorite subject in school and still is my favorite thing to read and learn about. (My grades were not a reflection of my passion- they were definitely not straight A's). I have taken my love of history and turned it into a school philosophy that works great for us.
A mutual friend of ours taught a class years ago titled, "History Really is Everything". It gave me such inspiration and I decided that could really be true. So, as Frankie described, we enjoy The Story of the World books and activities; as well as the accompanying literature. I also like to go a little further. We like to see who our ancestors are from that area of the world, or that period in history. It was interesting to learn about our ancestor who lived during the time of Henry VIII and imagine what his life must have been life in England. We even went on google earth and looked for the town where he was from. We also like The New Dictionary of Cultural Literacy; we have a corresponding literacy bee and test one another in our cultural literacy of the given time period or area of the world we are studying. It is very fun and there is no pressure; all of the kids enjoy it. Music and art can also be explored...we are just delving more into this area. Mathematics, philosophy, astronomy and language are also fun tangents to explore.
We are also learning to appreciate the Genevieve Foster books. We find they are a terrific companion to The Story of the World series. We have tried timeline's, but they do not seem to work well for our family. Maybe they would for yours...they are worth a try.
I agree with Frankie's book recommendation. Our family has a library of over 2,000 books and most of them were acquired from our local thrift store. The library is also a great option; they often have book sales which beat the thrift store hands down. Paperbackswap.com is a great option as well. We have more than one copy of some titles because they have become such classics it has been necessary to replace them or read them together. Books are a treasure! If you cannot purchase even used books now, as I have been unable to do at some points in our homeschooling life, the internet can be a great resource. Google books has a great repository of out of print, public domain books; many of which are classics you will want for your library. There are also other sites you can search for which feature books which are in the public domain and therefore, free of copyright. We found a very quaint science text called The Story of Science which approaches science from a story perspective; it is outdated, but enjoyable, and the truths are still...well, true.
I believe that the most important books in a family's library are their histories. Not the world's history, but your family's history. The link my children have to their world through the ancestors who lived in it before them, is an immeasurable asset in their lives. You don't need any money to assemble those; just a little time and imagination. One great story about grandma or grandpa can be worth far more than the noblest moldy monarch in the child's mind and heart.
Now, instead of asking the Lord, "Homeschooling? Do I have to?", I ask him, "Homeschooling? Why doesn't everyone?"