It used to irritate me that explanations of modern physics usually fell into one of two categories: either they would be "simplified" to the point that they are too vague or distorted from the truth to contain meaningful information, or they would use so much jargon and fancy mathematics that people who hadn't spent years studying them would have no clue what it meant. However, it turns out that if you know where to look, you can find some amazingly good explanations out there that let you learn how modern physics actually works without having to learn lots of fancy jargon and mathematics! To save you the effort of finding them, here are my favorite ones I've found so far:
The 2 best physics theories that we know of so far are general relativity (describing gravity) and the Standard Model of particle physics (describing electromagnetism and the nuclear forces), but they both build on top of special relativity's description of transformations in space and time. There's a good number of fairly decent explanations of special relativity out there, but this one won the Pirelli Relativity Challenge, so it better be good. After that, you can get a feel for special relativity by playing Relativistic Asteroids and A Slower Speed of Light! (I'm so sad that some of these links are no longer available at their original locations.)
General relativity: (our most accurate theory of gravity)
Good non-mathematical explanations of general relativity are much harder to find, and the only one I've found so far that really does the job is Chapter 42 of Volume 2 of the Feynman Lectures on Physics by Richard Feynman. (You won't miss anything important if you skip the mathematical parts.) It stops short of explaining more exotic concepts such as black holes (which were not widely recognized to exist at the time the book was written), but that's OK because it puts explanations of such concepts in the right context. Also, if you have Six Not-So-Easy Pieces, it's worth noting that the last chapter of Six Not-So-Easy Pieces is exactly the same as this chapter in the Feynman Lectures on Physics.
QED: (our most accurate theory of electromagnetism)
If I had to pick one thing from this list that I recommend reading, it would be QED: The Strange Theory of Light and Matter by Richard Feynman. This is a book that manages not only to describe quantum mechanics in a not-too-confusing way, but that gives a nearly complete description of our most accurate quantum mechanical theory of electrons and light, without using a single mathematical equation*, in enough detail that you could make precise predictions with it**, including lots of real life applications, all in only 150 pages! If that isn't enough, if you're not willing to pay $10 for the book (which it is absolutely worth), the original lectures that the book was based on are available to watch online for free! (and also on YouTube)
* The book contains a few simple mathematical expressions (which are not technically equations), but they are few and far between and not the emphasis of the book.
** The predictions would be very precise, but since the book stops just short of explaining spin polarization in detail, those predictions would not describe particles that actually exist in nature, which is a shame because the book comes so close to letting you do that. If that bothers you, perhaps the book will inspire you to read the Feynman Lectures on Physics to learn this remaining detail of "spin," using fancy mathematics to let you do calculations faster!
The rest of the Standard Model: (our most accurate theory of all known fundamental forces except gravity)
As complicated as our most accurate theory of electromagnetism may be, it is actually a relatively simple theory compared to the ones describing the strong and weak nuclear forces. Therefore, rather than learning the probability of each possible interaction to occur, it is perhaps better to settle for just knowing what those possible interactions are. The best reading I've found for doing that is Flip Tanedo's blog series about Feynman diagrams. (This first post I've linked to links to the remaining posts.) I think there are some places where it could explain things more clearly (e.g., it conflates Goldstone bosons and Higgs bosons by calling them both the Higgs, which I find really confusing), but as far as I can tell this series is one of a kind in what it does. (If you're simply interested in the Higgs boson, this video series does a nice job of explaining it.)
Here is by far the most to-the-point explanation I've found of what quantum computers do faster than classical computers, and the limitations on the mechanism they use.
From reading the QED book, it is clear that quantum mechanics is weird, but it may not be clear exactly what people find weird or unsettling about it. Given that, I figured it's worth linking to this well written explanation of Bell's theorem, which is a mathematical proof about a process called quantum entanglement that essentially quantifies what people find spooky about quantum mechanics. This concept is also heavily exploited by quantum computers.
Deriving quantum mechanics:
Here is an explanation of why it makes sense that the basic probability rules of quantum mechanics are what they are. This is part of an enjoyable (if technical in later parts) series of lecture notes that connect quantum computing to the wider intellectual world.
History of quantum field theory:
The Infinity Puzzle by Frank Close is not only one of the better physics books aimed at laypeople, but probably the definitive historical account of quantum field theory for professionals as well. Parts of it also help clarify some of the more confusing sections of Flip Tanedo's blog series, though it isn't a replacement for it. It is worth noting that this book focuses only on the later parts of the story when the Standard Model really came together, rather than starting from the first forays into quantum mechanics. Since another book I have which tells the story of quantum mechanics from the beginning has to explain the first interpretations of quantum mechanical behaviors then undo them as people came up with better ones (which can be confusing to read at times), perhaps that is a good thing.
Last but not least, if you're wondering why I still haven't mentioned string theory after all this time, The Trouble With Physics by Lee Smolin will tell you why.