It is now widely agreed that a positive affective state is a crucial component of animal well-being. The judgment bias test represents a widespread tool used to assess animals' optimistic/pessimistic attitude and to evaluate their emotional state and welfare. Judgment bias tests have been used several times with dogs (Canis familiaris), in most cases using a spatial test with a bowl placed in ambiguous positions located between a relatively positive trained location (P) which contains a baited bowl and a relatively negative trained location (N) which contains an empty bowl. The latency to approach the bowl in the ambiguous locations is an indicator of the dog's expectation of a positive/negative outcome. However, results from such tests are often inconclusive. For the present study, the judgment bias test performance of 51 shelter dogs and 40 pet dogs was thoroughly analysed. A pattern emerged with shelter dogs behaving in a more pessimistic-like way than pet dogs. However, this difference between the two populations was detected only when analysing the raw latencies to reach the locations and not the more commonly applied adjusted score (i.e. average latency values). Furthermore, several methodological caveats were found. First of all, a non-negligible percentage of dogs did not pass the training phase, possibly due to the experimental paradigm not being fully suited for this species. Second, results showed a high intra-dog variability in response to the trained locations, i.e. the dogs' responses were not consistent throughout the test, suggesting that animals may not have fully learned the association between locations and their outcomes. Third, dogs did not always behave differently towards adjacent locations, raising doubts about the animals' ability to discriminate between locations. Finally, a potential influence of the researcher's presence on dogs' performance emerged from analyses. The implications of these findings and potential solutions are discussed.