I decided to cut out most of the technical stuff, and go for short and simple. There are lots of technical articles about genetic genealogy, and they can explain it just as well, or better, than I can. See the bottom of the post for links.
So just how reliable is genetic testing for genealogical purposes? Well it depends on what you are after. Are you looking to establish a close relationship with someone from 600 years ago? Don't count on it. Are you looking to see if you have markers indicating that some of your ancestors came from a certain region or belonged to a certain group, more likely, but it is not an ironclad guarantee that you are actually from that region, or belong to that group. If you are looking to establish a relationship in the past few hundred years, that is possible, but the closeness of the relationship will not be included, i.e. you will know you are related, but not how closely. To establish closeness, you need a paper trail. On the other hand a genetic test can rule out relationship.
How does it work? It is actually quite straight forward. Our physical sex is determined by whether or not we have two X chromosomes (women), or an X and a Y (men). The Y chromosome is passed down through the male line exclusively. Women are a little different, as the mitochondrial DNA must be tested. The mitochondria are basically cells within a cell, and they provide energy to the cell. At any rate, mitochondrial DNA (mtDNA) is transmitted from the mother to all the children. Both the Y chromosome DNA and mtDNA change at relatively constant rates, which allows us to test and determine a very broad level of closeness. Additionally, because the Y chromosome only follows the male line, and the mtDNA follows both lines, we can test for paternal and maternal origins.
For example, Y DNA changes at the rate of 1 mutation every 500 years on average, so you may find that you are related to someone sometime in the past 500 years, but it doesn't tell you how closely you are related, just that you are. If the test finds no commonality, then you are definitely not related. In addition, remember that the times span of the change is an average. The Y DNA may have changed between you and your father, or not have changed at all in the past 1000 years. We just don't know.
What happens if you get a negative result? There may have been an adoption. There may have been an illegitimate child. There may have been an error in the test. There may have been a name change. There may have been infidelity. Be prepared, none of us are (or were) perfect, and you may not like the answer you find.
So, I come back to the original question, is genealogical DNA testing reliable? The answer is yes, but within limits. It can determine if there is a relationship, but it cannot determine the closeness of the relationship. Alternately, it can prove that no relationship exists. Taken along with paper records it can be a valuable addition to your research. Genealogical DNA testing can also be used to help determine deep ancestry, in other words where in the world your ancestors are (mostly) from. Finally, remember there are no absolutes. Even the best genetic testing comes with a percent level of certainty, even if the level is 99% certain, there is always that 1% chance that the correct results are alternate to the main result.
If you do decide to have some sort of genealogical genetic testing done, look around, some groups will pay part or all of the test costs depending on your surname or ancestry.
There are lots of sources about genetic genealogy. Here are some I used:
The Devon DNA Project - you will need a paper trail leading back to Devon in order to be allowed to submit a DNA sample to this project.