REVIEW ARTICLE
Decision aid for the use of additional tests during the pre-employment medical examination (PEME) of seafarers
Alf Magne Horneland, Suzanne Louise Stannard
Haukeland University Hospital, Department of Occupational Medicine, Norwegian Centre for Maritime and Diving Medicine, Bergen, Norway
✉
Alf Magne Horneland, MD, Norwegian Centre for Maritime and Diving Medicine, Haukeland University Hospital, P.O. Box 1400, 5021 Bergen, Norway, e-mail: alf.magne.horneland@helse-bergen.no
Abstract
No laboratory tests and imaging techniques are recommended for routine use in the ILO/IMO Guidelines on the Pre-Employment Medical Examination (PEME) of Seafarers that form the basis for statutory certification. However, they are widely used as components of the PEME protocols developed by insurers, employers and national maritime authorities in an attempt to predict and reduce the risks from illness whilst working at sea. This may be justified on scientific, safety, economic or professional grounds. We propose a rational approach for deciding if and when tests can be justified for routine use in assessing a seafarer’s fitness for work at sea. This is based on well-established methods for determining the validity of screening tests in public health as well as the seafarer demographics. We do not address the well-established use of similar tests where illness is suspected but only when they are used for routine PEME screening of all seafarers.
(Int Marit Health 2017; 68, 2: 90–98)
Key words: laboratory tests, imaging techniques, pre-employment medical examination, seafarers
INTRODUCTION
Guidelines on the medical examination of seafarers, published by the International Labour Organisation (ILO) and the International Maritime Organisation (IMO) [1], recommend that seafarers should not be routinely required to have multiple biochemistry, haematology or imaging studies. The use of such tests should be limited to situations where there is a clinical indication or a well-defined excess incidence of a serious disease such as tuberculosis in the seafarer’s home population. Nevertheless protocols for pre-employment medical examinations (PEME) developed by some national maritime authorities or by insurers, shipping agents or companies and by crewing nations do include such tests [2]. A review of 19 different PEME protocols reveals that many tests are used, some more commonly than others (Fig. 1).
Figure 1. Number of pre-employment medical examinations (PEME) protocols using a variety of laboratory tests for screening
This article proposes a rational approach for deciding if and when such tests can be justified for routine use in assessing a seafarer’s fitness for work at sea.
BACKGROUND
The overall aim of the PEME is to reduce the risk of illness and/or safety problems whilst the seafarer is working at sea. These risks are prioritised differently in different PEME protocols depending on the perspective of the organisation specifying the contents of each protocol [3]. Priorities will differ based on the relative importance of personal and vessel safety risks, the risk to ship operations, the financial risk for the shipping company and insurer, the health risks to the individual and to others and the reputation of the flag. Priorities will also depend on the approach to risk management and loss prevention taken by the shipping company, crewing agent and insurer.
Qualitative diagnostic tests are normally used in medical practice to establish a diagnosis, and the test result is either “positive”, indicating a specific condition, or “negative”, indicating the absence of the condition. Quantitative tests can be used to establish the seriousness or stage of a condition, but sometimes also for the purpose of establishing a diagnosis. In the latter case, a cut-off point needs to be defined, above or below which further action needs to be taken.
However, having a medical condition is not the same as being unfit for service at sea.
For example, in a seafarer with hypertension, the level at which one would start treatment is lower than the level at which a seafarer would be seen as unfit to go to sea. National guidelines such as those published by the National Institute of Health and Care Excellence in the United Kingdom [4] suggest that a blood pressure of greater than 140/90 mm Hg requires further investigation and possible treatment. This is aimed at reducing the long-term risks of both acute hypertensive illness and the development of other conditions for which hypertension is a known risk factor, e.g. myocardial infarction and cerebrovascular attacks. However, statutory PEME guidelines suggest that a seafarer is fit to go to sea as long as the blood pressure is less than 160/100 mm Hg. The PEME examination is looking at the likelihood of an incident occurring in the next 2 years, not throughout the seafarer’s lifetime. Dependent upon the measurement(s) taken at the PEME, decision trees are available to assist the doctor in deciding on permanent/temporary unfitness or the use of a limited certificate to minimise the risk to the seafarer of developing complications whilst at sea and also to reduce the risk to the ship and shipowner of such an incident occurring [5, 6].
When laboratory tests are used in an algorithm that “rules in” or “rules out” a seafarer from service at sea, based purely on the presence of a disease or risk factor as measured by the test, we risk putting healthy employees, i.e. those who have almost no increased risk of ill-health whilst working at sea, on shore. This could be regarded as unjustifiable discrimination and an unethical process because it prevents them from working [7]. Equally, if inappropriate tests are used, we also risk sending a person who does have an excess risk of ill-health to work at sea, and increasing the exposure to risk for the seafarer, ship, shipping company and insurer.
RISK ASSESSMENT IN PEME
The importance of the identification of a medical condition or diagnosis lies in the probability of a medical incident linked to that diagnosis occurring whilst at sea. The “careful clinical assessment and analysis” required by the IMO/ILO guidelines [8] implies that an individual risk assessment is carried out to determine the risk of a seafarer suffering a complication or acute deterioration of a known medical condition, or developing a new medical condition based on known risk factors during the validity period of the medical certificate. Screening asymptomatic individuals for the presence or absence of a disease or risk factor may be part of the risk assessment and can be done by clinical examination alone and/or may require the use of further tests.
SCREENING TESTS IN THE GENERAL POPULATION
Some of the laboratory tests and imaging techniques that have been used during seafarer PEMEs are essentially tests that are more widely used for diagnostic purposes based on the suspicion of a disease. Few, if any, have been shown to be suitable for screening of asymptomatic members of a particular group within the general population in order to identify undetected diseases or disease risk factors, in accordance with accepted criteria for evaluation of screening tests [9]. The criteria for a beneficial screening programme and screening test that focuses on detecting disease at an asymptomatic stage to facilitate secondary prevention is shown in Table 1. Well established population screening techniques include blood pressure monitoring in the identification of hypertension, mammography for the detection of breast cancer and cervical smears for the detection of cervical cancer. A wide range of other tests have been proposed for use in screening, but their validity can often be controversial e.g. prostate specific antigen for prostatic cancer detection.
Table 1. Criteria for screening programmes and screening tests used for detecting disease for the purpose of secondary prevention [9]
|
For a screening programme to be beneficial the disease in question should be
|
|
In addition the test itself should be
|
SCREENING OF ASYMPTOMATIC SEAFARERS IN PEME
The requirements for a screening programme focusing on identification of disease for the purpose of secondary prevention are not directly applicable for use as screening tests in PEME. The purpose of PEME is not to find seafarers with asymptomatic disease that could be treated to ensure a better outcome for the individual, but rather to identify seafarers with an excess probability of having a medical incident during the validity period of the certificate that could impose a risk to crew, ship and self whilst serving on board. Screening for the purpose of identifying asymptomatic seafarers that should not go to sea, requires different criteria, particularly with respect to the medical condition (Table 2).
Table 2. Suggested criteria for screening of asymptomatic seafarers
|
Disease criteria:
|
|
In addition the test itself should be
|
The disease in question should be common and its natural progression with or without treatment should possibly cause a significant medical problem within the validity period of the certificate. Furthermore, the medical problems and incidents associated with the condition may pose a risk (health/safety/financial/operational) to the seafarer, ship, shipping company or insurer. The test used also needs to be appropriate for detection of the disease or risk factor in an asymptomatic individual. Furthermore it needs to be safe and reasonably priced, with any follow up investigations or treatments widely available at a cost that is not prohibitive to the seafarer. Most importantly, use of a particular test in an asymptomatic population of seafarers should be shown to decrease the incidence of illness associated with that medical condition or risk factor and therefore decrease the risk to the seafarer, ship, shipowner and insurer. That is, it will improve risk assessment and risk management.
A SUGGESTED STEPWISE APPROACH TO THE APPROPRIATENESS OF A SCREENING TEST IN ASYMPTOMATIC SEAFARERS
Identification of conditions that cause medical problems at sea
Only conditions or risk factors that could lead to an unacceptable risk on board should lead to a decision of limitation, restriction or unfitness for service at sea. Therefore only those conditions should be considered for screening during PEME. If present, the likelihood of a medical incident occurring on board within the validity period of the certificate must be assessed. ’Risk’ in this context is the product of probability/likelihood and consequences. This means that if the consequences are catastrophic, even a low likelihood would lead to a high risk, and conversely, if the likelihood is very high, even smaller consequences could lead to an unacceptable risk.
Identification of such a disease or risk factor will allow further investigation and treatment of that disease to optimise the medical condition of the seafarer and hence maybe allow him to continue work without posing an increased risk on board.
Risk mitigation made possible through detection of disease
Will the detection of the disease or risk factor by the use of biochemical tests or imaging techniques make it possible to mitigate the risk imposed by the condition? Such mitigation could be restriction or limitation of medical certificates, decisions on temporary or permanent unfitness, the use of medication, surgery, glasses, hearing aid, pacemakers or setting other specific conditions for a certificate.
Capability of the test in detecting the disease
Prevalence. In screening programmes, a certain prevalence level is usually required to justify screening the general population in order to find cases suitable for secondary prevention. This is mainly due to cost-benefit analysis, but also to increase the positive predictive value of using the test. The higher the prevalence, the better the positive predictive value, and the lower the negative predictive value, and vice versa. To ensure a proper interpretation of test results, we need to know something about the prevalence in the population of concern.
Prevalence is the proportion, or percentage, of people with the disease in the population examined at a given point of time, or how common a disease is within a population. All cases are included in the numerator, and the denominator is the total population [11, 12]. Prevalence varies between different groups of people. This is a challenge in a system where seafarers are examined in many countries around the world, according to internationally applied standards and criteria, whilst prevalence rates may vary greatly between countries.
Prevalence also depends on whether you are looking at a non-selected group such as new seafaring recruits, or a group, such as serving seafarers, who have previously been selected.
If there are no old people, no infants, no obese or very many obese individuals in the study group, the prevalence data for some conditions will change. This will also be the case for seafarers who have PEMEs at regular intervals, as some then cease to work at sea because of medical problems found at previous examinations. We rarely have prevalence data for the specific group of people from which seafarers are recruited and this makes the assessment more difficult, and less accurate. However, as all individuals showing up for a seafarer medical have signed a self-declaration and will undergo a medical examination including a medical history, the pre-test likelihood will be influenced by this information and with less uncertainty than in the reference population. It is also important to note that as prevalence is the number of individuals with a certain condition at a certain point of time, old studies may not reflect the current frequency of disease.
The reference population. The reference intervals for a specific biological test vary with the population tested. Preferably this population should be as similar as possible to the population on which the test is going to be used, except for the presence of disease [11, 12]. This means that we ideally should have a reference population of candidates showing up for a PEME. The reference values will not necessarily be the same in different populations, and the reference intervals for a given test applied on people seeking employment at sea would vary in different parts of the world. In most cases we do not have such an ideal reference population, which means that validation of tests used in PEME will be more difficult, as general population data will have to be used as a reference.
The reference interval. Reference intervals for laboratory tests or imaging studies are set either according to statistical parameters or based on reference interval studies, data from the literature or the manufacturer, data mining studies or data from other centres. Usually the central 95% of a reference population is regarded as the reference interval, with a few accepted variations [10, 11, 12]. This implies that the lower 2.5% and the upper 2.5% of the test results outside the range are regarded “positive”, whilst those within the range are “negative”. On average one healthy individual out of 20 will have a test result that is defined as positive i.e. a false positive test. The use of multiple tests in a PEME programme or repeat results from the same test increases the accumulated number of individuals with false positive test results for each subsequent test added. The use of many tests without clinical indications and without applying discretion to the interpretation of the tests has the potential to unnecessarily reject healthy seafarers for work at sea.
Test sensitivity and specificity. Sensitivity is the proportion of people with disease who will have a positive test result [10, 12], or said in another way, the ability to find the individuals with disease.
If the test has 100% sensitivity, it means that it will identify all individuals who have the condition for which we are testing. If the test is negative, we can then be sure that the individual does not have the disease. In Figure 2 the test has a sensitivity of 80% so 8 out of the 10 people with the disease are correctly identified by this test. The lower the sensitivity, the more people with the condition will not be found. Relying on the test results from a test with a low sensitivity could lead to seafarers with the disease being falsely regarded as fit and sent to sea.
Figure 2 Test sensitivity: the proportion of people with disease that have a positive test result
Specificity is the proportion of people without disease who will have a negative result [10].
If the specificity is 100%, all individuals without disease will show a negative test. This means that if the test is positive, it is certain that the individual has the condition tested for. The example in Figure 3 shows a situation where the tests identify correctly 85 out of the 90 who do not have the disease, meaning that the specificity is 85/90 = 94.4%.
Figure 3. Specificity: the proportion of people without disease that will have a negative result
The lower the specificity, the less reliable is the positive test result. With a low specificity, a false conclusion of unfitness could be the result, putting healthy seafarers on shore, when they actually could have worked at sea.
Sensitivity and specificity are characteristics of the test used, and cannot be used alone to estimate the probability of disease in a single individual [10]. Sensitivity only looks at people with disease, specificity only at people without the disease. Sensitivity and specificity do not provide information that a positive or a negative test result is reliable in a single individual. Different parameters are needed to do this. Other ways of analysing test results can do so.
The positive and negative predictive values. How reliable is a test result? The positive predictive value (PPV) is the proportion of people with a positive test result who have the disease, see Figure 4 where the PPV is 62%, which means that 8 out of the 13 with a positive test result actually are correctly identified. It is dependent on prevalence (see below), or more precisely, the pre-test probability in the specific population where the test is carried out. The PPV increases when prevalence increases. Therefore the PPV for a certain test carried out on a given population cannot be transferred to another population if the prevalence is different [12].
Figure 4. Positive predictive value: the proportion of people with a positive test result that have the disease
Given a sensitivity of 80% and a specificity of 94%; the PPV will vary from 40% when prevalence is 5% to 96% when the prevalence is 60%. The higher the prevalence, the more likely it is that a positive test result actually indicates the presence of the disease [10].
The negative predictive value (NPV) is the proportion of those with a negative test result who do not have the disease [10, 12], see Figure 5 where 87 individuals are identified as not having the disease, whilst this is true only for 85 of them. In this case the NPV is 85/87 = 97.7%.
Figure 5. Negative predictive value: proportion of those with a negative test result who do not have the disease
It depends on prevalence, more precisely the pre-test probability, and decreases as prevalence increases, exactly the opposite to the PPV.
Again you cannot use the NPV for a given test on a specific population without checking that the prevalence is the same. If it is not, another NPV must be calculated. In the example above, the NPV will go down from 99% when prevalence is 5% to 76% when prevalence is 60%. The higher the prevalence, the less likely it is that a negative test result is reliable [10].
In the discussion of whether the test should be included in the programme, these values are of importance to consider.
The cut-off points. The accuracy of a diagnostic test is described by the sensitivity and the specificity of the test.
Both of them are dependent on the chosen cut-off point between normal and abnormal. The higher the cut-off point, the higher the sensitivity, but the lower the specificity.
Laboratory tests and imaging techniques currently used in PEME protocols have not been developed for the assessment of a risk that might occur at sea when a certain medical condition is present. Some conditions are easily treated on board; others lead to medical evacuation, diversion, hospitalisation, repatriation and replacement. The way we decide on the cut-off points is highly dependent on the objectives of the PEME protocols.
Test safety
When testing asymptomatic and therefore presumably healthy people with no indication of disease, it is important that the test itself does not impose a risk to the individual. Invasive techniques could have a potential of causing harm, as well as procedural deficiencies. It is in our opinion unethical to expose the individual seeking employment at sea to an individual risk of harm to his or her own health in the process of screening. We believe that potentially hazardous procedures should not be a part of a PEME examination.
Test costs
The cost of a test could be considerable and its widespread use may not result in a decrease in costs incurred due to a reduction of medical incidents occurring at sea. How many people must be screened using the test in order to ‘save’ the cost of one medical evacuation? In the clinical example of ultrasound scan (USS) for gall stones (Table 3), it will cost around £1,000,000 to avoid two medevacs due to a gall stone attack. The people who are paying for the USS are often not those who are paying for the evacuation costs, repatriation and replacement. In the assessment of the appropriateness of a test, the overall cost picture should be considered. If the cost of the many tests required is higher than the likely expenses that will be incurred if the test is not used, it should not be included in the PEME protocol unless it reveals an unacceptable risk whilst serving on board.
Table 3. Gallstones in the United Kingdom (UK) – statistical data
|
Consider a population of 10,000 seafarers in the UK |
|
Prevalence: 15% of UK population has gallstone disease according to the National Institute for Health and Care Excellence (NICE) i.e. 1500 of 10,000 seafarers |
|
The sensitivity of ultrasound scan (USS) for gallstones is 79% so of those 1500, USS will correctly identify gallstones in 1187 seafarers. Therefore 313 seafarers will be missed by USS and go to sea with their gallstones. |
|
The risk of symptoms occurring in asymptomatic patient with gallstones is 1–2% per year i.e. 2–4% over the 2 year validity period of the certificate [16–21] |
|
Therefore 3% of 1187 seafarers with known gallstones i.e. 36 seafarers will suffer symptoms related to gallstones over the next 2 years. Of these seafarers who develop symptoms, 5% will need hospital care [15] and therefore potentially evacuation and repatriation i.e. 2 seafarers. |
|
At the same time 1151 seafarers will be asymptomatic from there incidental gallstones and have been kept on land for no good reason. Some may have undergone surgery that would not be clinically indicated with the associated risk of the procedure itself plus the risk of complications e.g. adhesions over the validity period of their next certificate. Serious complications of endoscopic cholecystectomy occur in 2.6% [14] |
A CLINICAL EXAMPLE: ULTRASOUND SCAN OF THE GALLBLADDER IN ASYMPTOMATIC SEAFARERS
In the worked example in Table 3 we are putting 10,000 seafarers through an USS at a cost of approximately £100 per scan i.e. £1,000,000 to prevent two potential medevacs for hospital care over the next 2 years amongst the subgroup found to have incidental but asymptomatic gallstones.
In the meantime 313 seafarers will go to sea with gallstones not picked up on the scan and therefore despite this cost and of these 10 (9.4) may develop symptoms and 1 (0.5) will require hospital treatment and possible evacuation and repatriation.
Put another way we must screen 5,000 UK seafarers and recommend elective cholecystectomy in 593 seafarers to save symptoms occurring whilst at sea in the next 2 years in 18 seafarers and a potential medevac in one case.
The prevalence of gallstones is higher in Western countries than in others [13], so in countries with a lower incidence of gallstones, e.g. Philippines, it is likely that even more seafarers would need to be screened (and potentially stopped from going to sea) in order to prevent one medevac.
Test availability
The test should be available wherever the seafarers are having their PEME to ensure fair and equal assessment of the seafarers. The need for referral to a specialist make the test less useful and possibly less likely to be used, which reduces the relevance of including the test in a test programme.
Improved risk assessment
This is the salient point in the assessment. The overall objective of the PEME is to reduce the incidence of medical incidents at sea imposing a risk to crew, ship and self. Incidence data are available only to a limited extent, and structured evaluation of reduction of risk at sea by the use of screening tests in PEME, is to our knowledge, not yet published. Even if the tests cannot be properly validated statistically due to a lack of background data, some conclusions on the appropriateness can be reached through the structured process of assessing the tests.
PROPOSED FLOWCHART FOR A DECISION AID TO ASSESS THE APPROPRIATENESS OF THE USE OF SCREENING TESTS IN ASYMPTOMATIC SEAFARERS
Table 4 presents decision aid for the use of additional tests during the PEME of seafarers.
Table 4. Suggested decision aid for the assessment of the appropriateness of biochemical tests and imaging techniques in pre-employment medical examinations (PEME)
|
Step |
Description |
|
1 |
Identification of disease with the likelihood of unacceptable risk to ship, crew and self – and which can be identified by a specific test
|
|
2 |
Which test is considered to screen asymptomatic individuals? |
|
3 |
Which mitigating measures can a positive test result lead to? |
|
4 |
Is the test considered capable of detecting the disease at an asymptomatic stage?
|
|
5 |
Is the test safe? |
|
6 |
|
|
7 |
Is the test available to all seafarers internationally without need for further referral? |
|
8 |
Can use of the test lead to a demonstrable improvement in risk assessment? |
|
9 |
CONCLUSION: Is the screening test appropriate to manage risk to an acceptable level? |
DISCUSSION
The lack of consensus discussions between the different parties with regards to the level of risk that can be tolerated has led to the development of numerous PEME protocols with a different number and combinations of screening tests. Therefore, a seafarer can receive a different decision on fitness to work at sea from different protocols. A series of questions arises in this respect. Which diseases are necessary to avoid? Which diseases can be accepted? Which tests are useful and which are useless, for risk assessment? To what extent can seafarers be allowed to work at sea if disease risk factors are present or an asymptomatic diagnosis has been made? How can we use limitations in time and restrictions in sailing area and position on board in an evidence-based way to mitigate risk related to underlying medical conditions? Can we agree on the objectives for the PEME? Why do we need different protocols? How can we do this in an ethical way?
The questions and answers must be considered from the perspective of maritime authorities, seafarers, shipowners and insurers. Using the example above, the seafarer may well feel that a risk of less than 1% of symptomatic gallstones over the next 2 years is a risk worth taking compared the alternative of paying for an USS and potentially having to undergo elective surgery with its own risks, whilst he is losing money as he is unable to work during this time. On the other hand that risk of symptoms from gall stones and the chance of a medevac maybe too high for a shipowner or insurer at a time when margins are tight and any preventable interruption to operations and loss of productivity is not acceptable.
In medical conditions and screening tests with a variable scale rather than a yes or no answer the situation becomes even less clear. Ideally if such a test is to be used there should be a clear cut off point at which the seafarer is fit or unfit for work at sea. This may or may not be the same cut off point at which treatment for that condition is required.
The tests used to screen asymptomatic seafarers in PEME should preferably be validated for their use in this context. Important data for such validation, such as reference population data on prevalence and incidence are lacking, and data from the general population will have to be used. Currently there is no agreement on the appropriate cut off points for the tests being used as far as we are aware and there is no agreement on the level of risk that can be accepted. This makes it difficult to assess the appropriateness of the different tests used in PEME. As part of the process of deciding whether or not to adopt a test as part of a PEME examination there must be a discussion, and hopefully agreement, on the level of risk and therefore potential cost that can be tolerated by the different parties.
CONCLUSIONs
Despite the recommendation from the UN Agencies that laboratory tests and imaging techniques should not be used in the absence of an individual clinical indication, they are widely used for screening prior to employment on board ship. Whilst some may be a useful risk management tool in the assessment of seafarers others may offer no reduction in risk at sea whilst at the same time incurring significant costs for those paying, risks to seafarers of treating asymptomatic conditions or putting healthy seafarers ashore. Discussions are needed between the parties responsible for specifying and for working within different PEME protocols to seek consensus. Because of differences in prevalence, some screening tests may be useful in some populations but not others and hence recommendations should be tailored to the local need in the different areas where seafarers are recruited. There is also a need to develop criteria for use when new techniques are being considered. The stepwise approach suggested in this article provides a starting point for discussion and review of the use of such tests. However, a holistic individual risk assessment that is based on all the available valid information collected during the PEME and performed by a health professional with competence in maritime medicine should form the basis for decisions on the certification of fitness in all PEME protocols.
CONFLICTS OF INTEREST
No financial and material support. The authors are employed by the Norwegian Centre for Maritime and Diving Medicine and through this employment contracted as medical advisers to the Norwegian Maritime Authority.
REFERENCES
1. Guidelines on the medical examination of seafarers, Ch. XII, Paragraph (x). International Labour Office, Geneva, 2013. ISBN 9 789221 274629. http://www ilo org/wcmsp5/groups/public/@ed_dialogue/@sector/documents/normativeinstrument/wcms.
2. Personal review of PEME protocols from Panama, Marshall Islands, Malta, Liberia, Norway, the Netherlands, Philippines DOH, Steamship, The Standard, The American Club, UK P&I Club, The Swedish Club 2016.
3. Carter T. Seafarer medicals: population health or private gain? Int Marit Health. 2016; 67(1): 1–2, doi: 10.5603/IMH.2016.0001, indexed in Pubmed: 27029921.
4. https://www.nice.org.uk/guidance/cg127.
5. https://www.gov.uk/government/publications/the-approved-doctors-manual.
6. https://handbook.ncmm.no.
7. Universal Declaration of Human Rights, adopted by the United Nations General Assembly 10th December 1948, Article, 23: 1.
8. ILO/IMO Guidelines on the medical examination of seafarers. Appendix E. Introduction. ISBN 978-92-2-127463-6.
9. Herman C. What makes a screening exam. Virtual Mentor. 2006; 8(1): 34–37, doi: 10.1001/virtualmentor.2006.8.1.cprl1-0601, indexed in Pubmed: 23232314.
10. Akobeng AK. Understanding diagnostic tests 1: sensitivity, specificity and predictive values. Acta Paediatr. 2007; 96(3): 338–341, doi: 10.1111/j.1651-2227.2006.00180.x, indexed in Pubmed: 17407452.
11. Graham J, Barker A. Reference Intervals. Clin Biochem Rev. 2008 Aug(Suppl 1): 93–97, doi: PMID: 18852866. PMCID: PMC2556592.
12. Principles of Epidemiology in Public Health Practice, Third Edition, An introduction to Applied Epidemiology and Biostatistics. http://www.cdc.gov/ophss/csels/dsepd/SS1978/Lesson3/Section2.html Accessed 2016-09-23. (Accessed 2016-09-23.).
13. Njeze GGE. Niger J Surg. PMC:3899548. 2013; 19(2): 49–55, doi: doi:10.4103/1117-6806.119236.
14. Friedman GD, Raviola CA, Fireman B. Prognosis of gallstones with mild or no symptoms: 25 years of follow-up in a health maintenance organization. J Clin Epidemiol. 1989; 42(2): 127–136, doi: 10.1016/0895-4356(89)90086-3, indexed in Pubmed: 2918322.
15. Thistle JL, Cleary PA, Lachin JM, et al. The natural history of cholelithiasis: the National Cooperative Gallstone Study. Ann Intern Med. 1984; 101(2): 171–175, indexed in Pubmed: 6742647.
16. Shabanzadeh DM, Sørensen LT, Jørgensen T. A Prediction Rule for Risk Stratification of Incidentally Discovered Gallstones: Results From a Large Cohort Study. Gastroenterology. 2016; 150(1): 156–167.e1, doi: 10.1053/j.gastro.2015.09.002, indexed in Pubmed: 26375367.
17. Afdahl, N H: Approach to the patient with incidental gallstones. www.UpToDate.com, literature review current through Sep 2016. https://www.uptodate.com/contents/approach-to-the-patient-with-incidental-gallstones?source=search_result&search=approach%20to%20the%20patient%20with%20incidental%20gallstones&selectedTitle=1~150 (Accessed 2016-10-03).
18. Attili AF, De Santis A, Capri R, et al. The natural history of gallstones: the GREPCO experience. The GREPCO Group. PMID: 7875663. Hepatology. 1995; 21(3): 655–660, indexed in Pubmed: 7875663.
19. Capocaccia L, the GREPCO group. Clinical symptoms and gallstone disease: Lessons from a population study. In: Epidemiology and prevention of gallstone disease, Capocaccia L, Ricci G, Angelico F, Attili AF (Eds), Lancaster MTP Press, 1984. p.153. http://link.springer.com/chapter/10.1007%2F978-94-009-5606-3_20#page-2.
20. Barbara L, Sama C, Morselli Labate AM, et al. A population study on the prevalence of gallstone disease: the Sirmione Study. Hepatology. 1987; 7(5): 913–917, indexed in Pubmed: 3653855.
21. Gracie WA, Ransohoff DF. The natural history of silent gallstones: the innocent gallstone is not a myth. N Engl J Med 1982; 307:798. http://www.nejm.org/doi/full/10.1056/NEJM198209233071305.
