So let's look at some data. These are some numbers I computed from pedigrees of several different dogs. One had very low inbreeding (< 5%), the others had significantly more. For each of these individuals, though, the way the computed COI changed with the number of generations of data used was different. One thing is clear - unless the level of inbreeding is very low, or the pedigree very shallow, using only 5 generations of pedigree data is likely to be extremely misleading.
In this figure, it's easy to see how far wrong you can go. One animal (pink) has a COI at 5 generations of about 6% (not bad); at 10 generations it is up to about 10% (still not so bad); but including 30 generations of data reveals that the true COI is about equal to what you would get from the mating of full siblings (25%).
If you're worried about the probability of producing puppies with a genetic disorder caused by a recessive mutation, there is a whopping difference between the relatively low risk of about 1 in about 20 (6% COI with 5 generations) and 1 in 4 (25% COI using 30 generations).
Does it matter how many generations you use when calculating COI? Yes. How many generations should you use? As many as you can.