BLACK BOOK METHODOLOGY
QUALITATIVE INSIGHT INTO THE INDUSTRY’S TOP VENDORS THROUGH 18 KEY PERFORMANCE INDICATORS
Black Book collects ballot results on 18 performance areas of operational excellence to rank vendors by software, systems, products, equipment and outsourced service lines. Client users submit ballots per vendor per function which are juxtaposed with client size, functions outsourced, software installed, systems employed, services rendered, industry/vertical(s) affected, country of service origin and overall service and/or product line domain.
The gathered data is subjected immediately to an internal and external audit to verify completeness and accuracy and to make sure the respondent is valid while ensuring the anonymity of the client company is maintained. During the audit, each data set is reviewed by a Black Book executive and at least two other external reviewers. In this way, Black Book report clients are able to clearly see how a vendor is truly performing. The 18 criteria on operational excellence are subdivided by the client’s industry, market size, geography and software category/function outsourced and reported accordingly.
UNDERSTANDING THE STATISTICAL CONFIDENCE OF BLACK BOOK DATA
Statistical confidence for each performance rating is based upon the number of organizations scoring the outsourcing service. Black Book identifies data confidence by one of several means:
- Top-10-ranked vendors and advisors must have a minimum of five unique clients represented. Broad categories require a minimum of 20 unique client ballots. Data that are asterisked (*) represent a sample size below required limits and are intended to be used for tracking purposes only, not ranking purposes. Performance data for an asterisked vendor’s services can vary widely until a larger sample size is achieved. The margin of error can be very large and the reader is responsible for considering the possible current and future variation (margin of error) in the Black Book performance score reported.
- Vendors, associates, consultants and advisors with over 20 unique client votes are eligible for top 10 rankings and are assured to have highest confidence and lowest variation. Confidence increases as more organizations report on their outsourcing vendor. Data reported in this form are shown with a 95% confidence level (within a margin of 0.25, 0.20 or 0.15, respectively).
- Raw numbers include the quantity of completed surveys and the number of unique organizations contributing the data for the survey pool of interest.