Contractor Scorecard Calculations
- Caroline Antoun
Contractor scores are calculated using data science techniques to provide the most statically significant and accurate scores that ServiceChannel can provide. This comes with increased complexity which you will not be able to easily average or add up the work order scores to calculate the overall scores. However, this article will break down at a medium level overview how scores are being calculated, which KPIs are heavier weighted, and what actions make up the KPIs.
Key Considerations for Score Calculations
Scores now have a numerical value and no longer use letter grades.
Scores are based on an absolute scale of performance, so scoring at or near 100 is impossible in the real world.
Scores in each performance category Speed, Quality, Engagement, and Price (US) are ranked across the entire ServiceChannel platform and ranked by quartiles: Top, Average, Below Average, and Bottom. These quartile calculations are dynamic.
Due to exclusions, the majority of work orders will not have a value for one or more of the underlying score KPIs. This does not negatively impact scores.
- Of the 3 years of calculated data, most recent data is given the heaviest weight.
How is Performance Scored?
Performance is broken down into four main categories: Speed, Quality, Engagement, and Price. Speed and Quality scores are based off of work orders. Engagement score measures a providers Fixxbook engagement activity. Price scores are for US customers only and are a dynamic aggregate of invoices and therefore, individual work orders are not scored for price.
Outside the US
If you are outside of the US, the Price score tabs and columns will still show, but you will see nulls in the score placement. This does not negatively impact scores.
Weighted Scores
In each category, there are specific KPIs that are scored and then totaled. Each KPI carries a specific weight calculation based off importance to each category. For example, for Speed, On Time Arrival is the most important factor for speed score vs Same Day Completion and Time on Site – the two lowest weighted KPIs because 1) the work order might not have required same day completion (in which it would not be scored for that) and 2) A provider taking a certain amount of time on site is not as impactful to work order completion as a provider not showing up in a timely manner.
Performance Categories and their KPIs
Here are the categories with each KPI ordered in terms of importance and heaviest weight given. You will note Price KPIs are not ranked because their calculations are much more dynamic than the other categories.
KPI Definitions and Compliance
In each performance category, individual KPIs that get met are scored and then tallied up into an overall score. Not every KPI will be scored and a NULL will be seen. This will not negatively impact the score – it simply won't be calculated with the total. For example, with the Speed score, you often won't have a score for Same Day Completion as a majority of your work orders might not have priorities that require same day. This KPI will not be scored on those work orders.
For a KPI to be scored, you need minimum 10 work orders within the selected time frame unless otherwise specified. To have an overall score, you need to have at least one KPI score. When a KPI has a NULL value, it will not be factored into the over all score.
| KPI | Definition | How it's Scored | Scoring |
|---|---|---|---|
| Same Day Completion | Whether the work order was completed within 24 hours | Only work orders that have a max ETA of 24 hours will be scored. | 100 points - Completion 0 - arrival > 24hrs |
On Time Arrival (Updated Scheduled date) | Whether the provider came on site before the updated schedule date | There is a Check-in, an "In Progress, Onsite" status, or Completion Date by the current Schedule Date. | 100 points - Onsite by Scheduled Date 50 points - Within a 30 minute window 0 points - Late NULL (not scored) - No check-in or completion date |
On Time Arrival (Original date) | Whether the provider came on site before the original ETA date | There is a Check-in, an "In Progress, Onsite" status, or Completion Date by the Scheduled Date. | 100 points - Onsite by Scheduled Date 80 points - Within a 30 minute window Scaled - More than 30 min late NULL (not scored) - No check-in or completion date |
Timely Dispatch Confirmation | How long the provider took to dispatch the work order | Time provider confirmed dispatch from the Call or Reassign Date. Must be within 75th percentile of peers to be considered compliant. | 100 points - Compliant 0 points - Non compliant |
Proposal Submission Time | How long the provider took to submit proposals | Time the WO was in "IN PROGRESS, WAITING FOR e QUOTE" status. For work orders with Priority up to 24 hours, 1,440 minutes or 1 day is considered to be on time. For WOs with Priority over 24 hours, 2,800 minutes or 2 days is considered to be on time. | 100 points - Compliant 0 points - Non compliant NULL (not scored) - If the work order never went into WAITING FOR QUOTE status. |
Provider Completion Time Without Proposals | How long the provider took to complete the work order without proposals | Time a provider was responsible for a work order based on the status/extended status combinations. Scoring is dependent upon cross platform data in Priority and Trade and broken down in percentiles. *Must have 30 work orders in that Priority and Trade. | < 10th% : 100 points NULL (not scored) - Less than 30 work orders in that trade/priority. |
Provider Completion Time With Proposals | How long the provider took to complete the work order with proposals | Time a provider was responsible for a work order based on the status/extended status combinations. The total time will be a sum of minutes spent in each status/ext status combination identified as owned by the provider. Scoring is dependent upon cross platform data in Priority and Trade and broken down in percentiles. *Must have 30 work orders in that Priority and Trade. | < 10th% : 100 points NULL (not scored) - Less than 30 work orders in that trade/priority. |
Time on Site | How long the provider spent on site | Total time that a work order was in status "IN PROGRESS, ON SITE", with each individual onsite status capped at 24 hours in case a provider forgot to check out. Scoring is dependent upon cross platform data in Priority and Trade and broken down in percentiles. *Must have 30 work orders in that Priority and Trade. | < 10th% : 100 points NULL (not scored) - Less than 30 work orders in that priority/trade. NULL (not scored) - Didn't check in NULL (not scored) - Less than 5 minutes on site. |
| Time to Site | How long the provider took from start date (call or reassign) to on site | Time difference between Call Date or Reassign Date to the first "IN PROGRESS, ON SITE" status. Scoring is dependent upon cross platform data in Priority and broken down in percentiles. *Must have 30 work orders in that Priority. | < 10th% : 100 points NULL (not scored) - Less than 30 work orders in that trade/priority. NULL (not scored) - Didn't check in |
For a KPI to be scored, you need minimum 10 work orders within the selected time frame. To have an overall score, you need to have at least one KPI score. When a KPI has a NULL value, it will not be factored into the over all score.
| KPI | Definition | How it's Scored | Scoring |
|---|---|---|---|
| One Time Visit | Whether the provider finished the work order in one visit | Work order was completed with only one "IN PROGRESS, ON SITE" status. Note: Will be penalized if technician checks in and out for lunch. | 100 points - Compliant 0 points - Non compliant NULL (not scored) - No "IN PROGRESS, ON SITE" status on this work order |
| Check-In | Whether the provider checked in or not | At the WO level, the Check-in Score indicates a 100 if the provider checked in and a 0 if the provider did not check-in. The aggregate Check-in Score across multiple WOs is a weighted score calculated based on platform comparisons of check-in scores by trade. | 100 points - Checked in 0 points - Did not check in |
| Recall Work Order | Whether the work order entered and then exited a completed status | The number of times that a WO moved out of a COMPLETED status. | 100 Points - Work order never exited COMPLETED status -1000 points - Work order exited COMPLETED status more than once. *Aggregated Recall Work Order score will never be lower than 0. |
| Recall On Linked WOs | Whether the work order was linked to a recall work order | -1000 points - Recall on Linked WO *Aggregated Recall Work Order score will never be lower than 0. |
Score is calculated on the fist Sunday of each month.
| KPI | Definition | Scoring |
|---|---|---|
| Active Technician | Number of technicians who check in the last one year | 100 points - 1 technician checked in in the last one year 0 points - 0 technician check-ins |
| Added Technician | Number of total technicians group by providers | 100 points - 1 technician added in the last one year 0 points - 0 technicians added |
| Invitation | Average scoring for response time between subscribers and providers. Within the past 1 year | 100 points - Invitation response time is less than 72hours 50 points - Invitation response time is between 72 hours and 30 days 0 points - greater than 20 day response time |
| Labor Line Item Invoice | Percent of invoices with labor line item
| Percentage based on percent of invoices that meet criteria. Ex 78% = 78 points |
| Material Line Item Invoice | Percent of invoices with materials line item(s)
| Percentage based on percent of invoices that meet criteria. Ex 78% = 78 points |
| Resolution Code (US only) | Percent of work orders with resolution codes filled in | Percentage based on percent of work orders that meet criteria. Ex 78% = 78 points |
| Technician Profile Completion | Based on: 1.Technicians' phone fill rate 2.Technician's picture fill rate 3.Average technician's phone and photo fill rate | Percentage based on average of phone fill rate and picture fill rate. Ex: Average is 85% = 85 points. |
| Compliance Completion | Average of compliance_percents | Percentage based on average compliance level across providers |
| New Provider Profile Completion | New Provider Profile Completion | Percentage based on percent profile completeness. Ex 68% complete = 68 points |
Price score is for US locations only due to the reliance on Core Based Statistical Analysis (CBSA) data that is defined by US Office of Management and Budget. Scoring is dynamic and comparative, and heavily reliant upon current marketplace data in the same trade and CBSA in order to be the most relevant.
| KPI | Definition |
|---|---|
| Median invoice amount without Proposals | The median invoice amount for work orders that didn't have proposals |
| Median invoice amount with Proposals | The median invoice amount for work orders that had proposals |
| Invoices < completed in less than 2 hrs | % of completed Invoices with 2 or less labor hours |
| Invoices > completed greater than 5 hrs | % of completed Invoices with 5+ labor hours |
| Invoice to NTE Comparison | Median Percentage difference vs. NTE |
