Grapes have proved to be one of the most cost-effective and profitable crops for cultivation in India. This crop however is affected by numerous diseases which cause significant yield losses every year. Early detection of diseases and proper identification of their severity will help to take decisions on proper usage of pesticides in terms of their type and quantity, which eventually will help in maintaining the crop health. In this work, we propose a system for classifying three diseases affecting grapes — Anthracnose, Powdery Mildew and Downy Mildew — and identifying the severity of these diseases using image processing and machine learning algorithms. The key contribution of the proposed system is to consider images of grapes leaves with complex background which are captured under an uncontrolled environment. We compare the performance of four machine learning algorithms, PNN, BPNN, SVM and Random Forest, for separating the background from disease patches and classifying between the different diseases. We also study the performance of different texture features like local texture filters, local binary patterns (LBP), GLCM features, and some statistical features in RGB plane for classification. The proposed system achieves best classification accuracy of 86% using Random Forest and GLCM features.