We are given the annotation for each image by the ophthalmologist/expert. Consider for e.g. that the annotation for Image1 is as given below:-
| 111111111111111111111111111111111111113 |
In this annotation there are 39 numbers, each number corresponding to one of the 39 manifestations. According to the above annotation, manifestation 39 is present in state 3. All other manifestations are present in state 1(absent or normal).
Our earlier attempts at diagnosis have not yielded 100% results. This led us to a study of the annotations given. We see that given an image if a particular diagnosis has to be made we require pointers indicating that particular disease. The ability of the system to diagnose correctly depends greatly on these indicators. If the annotations are such that they yield ambiguous indicators the probability of the system making a correct diagnosis is reduced. We need to find the ideal annotations for each diagnosis. These would be annotations that are clear indicators of the diagnosis being considered.
We ignore probability values in state 1. We run all these numbers through the Fischer's linear discriminant test. The Fischer's test is used such that a manifestation is annotated, for a particular diagnosis, as other than 1 if and only if any other state emerges as a clear indicator of that particular diagnosis.
For one particular diagnosis there might be a manifestation which is a clear indicator of the diagnosis in question, in more than one state in which case both the states are individually combined with the other annotations to yield 2 different ideal annotations.
This procedure is repeated for all the diagnoses.
| Manifs | Diag1 | Diag2 | Diag3 | Diag4 | Diag5 | Diag6 | Diag7 | Diag8 | Diag9 | Diag10 | Diag11 | Diag12 | Diag13 | Diag14 |
| 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| 2 | 1 | 1 | 1 | 1 | 2 | 2 | 2 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| 3 | 1 | 1 | 1 | 2 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| 4 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| 5 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 2 | 1 |
| 6 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| 7 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| 8 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| 9 | 1 | 1 | 1 | 1 | 1 | 1 | 2 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| 10 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| 11 | 1 | 1 | 1 | 4/5 | 1 | 1 | 6 | 1 | 1 | 6 | 1 | 1 | 1 | 1 |
| 12 | 1 | 1 | 1 | 1 | 1 | 1 | 2 | 1 | 1 | 2 | 1 | 1 | 1 | 1 |
| 13 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 2/3 | 1 | 1 | 1 | 1 |
| 14 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| 15 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 3 | 1 | 1 | 1 | 1 |
| 16 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| 17 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| 18 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 3 | 1 | 1 | 1 | 1 |
| 19 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 3 | 1 | 1 | 1 | 1 | 1 | 1 |
| 20 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| 21 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| 22 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| 23 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| 24 | 1 | 3 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 2 | 1 | 1 | 1 | 1 |
| 25 | 1 | 1 | 1 | 2 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| 26 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| 27 | 1 | 1 | 1 | 2 | 1 | 1 | 1 | 1 | 1 | 1 | 2 | 1 | 1 | 1 |
| 28 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 3 | 1 | 1 | 1 |
| 29 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 2 | 1 | 1 | 1 | 1 | 1 |
| 30 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 2 | 1 | 1 |
| 31 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| 32 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| 33 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| 34 | 1 | 2 | 3 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| 35 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| 36 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| 37 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| 38 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 4 | 1 | 1 | 1 | 1 | 1 | 1 |
| 39 | 2/3 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
The system used these annotations to make a diagnosis using three methods. The methods and the subsequent results are as follows:-
| Criterion | ||
| Method | Any-Match | Perfect-Match |
| Normalize-Sums | 100% | 94% |
| Noisy-Max-All Evidence | 100% | 64% |
| Noisy-Max-Annotated-Links | 67% | 57% |
The results tabulated by diagnosis are as follows(considering any-match only):-
| Diagnosis | Normalize-Sums | Noisy-Max-All Evidence | Noisy-Max-Annotated-Links |
| Diag1 | 100% | 100% | 100% |
| Diag2 | 100% | 100% | 100% |
| Diag3 | 100% | 100% | 100% |
| Diag4 | 100% | 100% | 100% |
| Diag5 | 100% | 100% | 100% |
| Diag6 | 100% | 100% | 0% |
| Diag7 | 100% | 100% | 50% |
| Diag8 | 100% | 100% | 0% |
| Diag9 | 100% | 100% | 100% |
| Diag10 | 100% | 100% | 100% |
| Diag11 | 100% | 100% | 100% |
| Diag12 | 100% | 100% | 100% |
| Diag13 | 100% | 100% | 50% |
The table and graph containig the 13 Diagnoses and their key indicators as found by the ideal annotations test are attached in these files.
The ideal annotations have helped us understand the workings of our diagnostic system. We have observed through all our experiments that of the 3 methods used for diagnosis the 'normalize-sums' and the 'NM-all evidence' methods have done much better than the 'NM-annotated links' method.
The possible reasons for this are:
1. The 'NM-all evidence' method considers all the manifestations ,this means that it does not look at only those manifestations that are linked or only those manifestations that are annotated as other than normal.Therefore in this method while making a diagnosis there is a weighing of all the evidenve given.In addition to looking at what is present in our attempt to diagnose we also look at what is not there.
For e.g. Diag1 has manifestation 39 in state 2 in its ideal annotation.The ideal-separable annotations tells us that in this state this manifestation is also an indicator of Diagnosis 2. Since manifestation 39 is the only one linked to Diagnosis 1, the system is likely to come up with a diagnosis of both 1 and 2 in the 'normalize sum' method (which is what happened).The system did not take into account that for a diagnosis of 2 to be made it also has to see manifestation 24 in state 3. This is not the case with 'NM-all evidence' method. Since all the manifestations are considered it comes up with only Diagnosis 1 as the right diagnosis. It has not only considered what it sees but also what it doesnt see.Thus it can be seen as a method that rewards presence of and penalises the absence of key indicators.
The 'NM-annotated links' method fails due to the same reasoning as above.
2.The 'normalize-sums' method does well as this method is more like a physicians thought process. when a physician considers the possibility of a diagnosis he does not and cannot have in mind all the 14*39*7(No. of diagnoses* No.of manifestations* No. of states)probability values. He is more likely looking at the importance of a manifestation relatively. That is given manifestation 3 is in state 2, he is more likely to compare what that information means for each of the diagnoses being considered. Is his belief of a diagnosis's presence strengthened or weakened by this information? As the 'normalize-sums' method so closely follows this ideology we have had such success with this method.
Appendix:-
Fisher's Linear Discriminant Test:-
The Fishers test is as follows: The 14 values(probability values) are sorted such that Ni<=Nj for all i,j. These sorted values are partitioned into 2 sets A and B with the first 13 elements into the first set and the last element into the second set. Then the mean and the standard deviation for both the sets are calculated and the Fishers criterion is found as ( m a - m b)*( m a- m b)/( s a* s a+ s b* s b)=Fp1 , where m a, m b are the means of the partitions and s a and s b are the standard deviations of the 2 partitions.
In succeeding iterations the size of set A is decreased and that of set B is increased. The largest Fishers criterion so found is considered as the breakpoint and all the diagnoses in set B are considered as the system diagnosis. Sometimes more than one diagnosis can emerge as the cause for the presence of the manifestations. There is a maximum limit of 3 diagnoses that can be so indicated. This is to ensure that the system makes a definitive diagnosis rather than one which indicates all the diagnoses in the database as the cause. The results of the Fishers test are the resultant system diagnosis.