All benchmarks are results of evaluations.
First, table 1 is created to help you organize your paper. It has the factor to be evaluated, evaluations and benchmarks columns. This table can be in the result section and you can organize your paper according to it. It is optional, so do not panic!
Table 1: EHR evaluation table
| FACTORS TO EVALUATE | EVALUATIONS & BENCHMARKS | |||||
| Description | Reference | Phase I Evaluation | Phase I Benchmark | Phase II Evaluation | Phase II Benchmark | |
| Factor | The game should be physically safe to play. | Yeweifale, 2021 | Systematically find every possible controller that could be used to input data into the game and test it with the game and with respect to in-game tips. | No (0) incompatible third party controller is found, or tips or software are modified to make third party controllers compatible. | Post-consumer reports of mishandled third party controllers must be systematically evaluated | Post-consumer reports of mishandled third party controllers should be at or below a rate of 0.1% of the number of game copies sold |
| Factor | ||||||
| Factor | ||||||
| Factor |
Issues 1: Factors listed in Evaluation part/ Listed factors are not detailed enough
I encouraged all of you to put the factors in a table, so you can make sure that you specify them properly. Multiple factors I reviewed had factors accidentally listed in the (especially Phase I). These all also had the problem that a factor wasn’t detailed. A factor should be what you want to analyze, and the evaluation is how you want to analyze it. Basically, the how and the what should be specified and detailed enough.
Examples (these aren’t from your options because I can’t give away answers):
Let assume that you include a table with all the requirements that you are going to discuss in this assignment:
Table 2: Issue related to the factors listed in the evaluation part/ listed factors are not detailed enough
| Factor (What) | … | Phase I Evaluation (How) | Phase I Benchmark | Phase II Evaluation (How) | Phase II Benchmark | |
| Correct | Patient portal systems need to be usable | … | An accepted systems usability scale is used to test the portal system with 5 patients | The average usability score is 75% | (Same as Phase I) | The average usability score is 90% |
| INCORRECT | Usability | … | Patient portal systems need to be usable | An accepted systems usability scale is used to test the portal system with 5 patients, average usability score is 75% | Patient portal systems need to be usable | An accepted systems usability scale is used to test the portal system with 5 patients, average usability score is 90% |
Issue 2: Phase I benchmarks are counterproductive
I also found that some misunderstood what the Phase I benchmarks should be like. If a Phase I benchmark has a Phase II benchmark for the same evaluation, Phase I should be less strict. Some misinterpreted this as that the Phase I benchmark being lenient to the point that something bad was being expected.
Examples:
Table 3: Issue related to the phase I benchmarks are counterproductive
| Factor | … | Phase I Evaluation | Phase I Benchmark | Phase II Evaluation | Phase II Benchmark | |
| Correct | Physician critical care dashboards need to be legible | … | Physicians are tested for % accuracy in reading all items on the dashboard | On average, the % accuracy must be at least 90% | (Same as Phase I) | On average, the % accuracy must be at least 98% |
| INCORRECT | Physician critical care dashboards need to be legible | … | Physicians are tested for % accuracy in reading all the items on the dashboard | On average, the % accuracy must be less than 98% | (Same as Phase I) | On average, the % accuracy must be at least 98% |
Issue 3: Evaluations not related to benchmarks
This issue often stemmed from factors being placed in the what to evaluate. However, it also came when the evaluation was placed and written correctly. All benchmarks are results of evaluations.
This example has a good evaluation, but the benchmark doesn’t directly relate to it. While record switching can cause patient mortality, finding the number of critical incidents due to switched records won’t give us a mortality figure to compare to a benchmark.
Table 4: Issue related to the evaluations not related to benchmarks
| Factor | … | Phase I Evaluation | Phase I Benchmark | … | … | |
| Correct | Patient record switching is a major safety issue | … | The number of critical incidents due to switched records is analyzed | The number of critical incidents is below 1 per 10,000 patient-visits | … | … |
| INCORRECT | Patient record switching is a major safety issue | … | The number of critical incidents due to switched records is analyzed | The patient mortality is less than 1 per 10,000 patient-visits | … | … |
Issue 4: Factor and evaluations not related
This type of problem is similar to the last one. One cause was, again, a factor sitting in the evaluation column. The other times, the evaluation made sense but for some reason didn’t match the factor.
Example (with a valid albeit irrelevant evaluation):
Table 5: Issue related to the factor and evaluations with no relation
| Factor | … | Phase I Evaluation | … | … | … | |
| Correct | Systems need to schedule staff so that they have enough downtime between tasks | … | Staff are surveyed to determine the number of times they weren’t given enough time to go from one place to another | … | … | … |
| INCORRECT | Systems need to schedule staff so that they have enough downtime between tasks | … | The system logs are checked daily to find out the rate of staff who are absent from work. | … | … | … |
Issue 5: Remedial measure stated where the evaluation (or benchmark) should be
In a few places, as an evaluation or benchmark, a remedial measure (how to fix the problem) was stated instead.
Example: Evaluation is a remedial measure
Table 6: Issue related to the remedial measure stated where the evaluation (or benchmark) should be
| Factor | … | Phase I Evaluation | Phase I Benchmark | Phase II Evaluation | Phase II Benchmark | |
| Correct | Physician critical care dashboards need to be legible | … | Physicians are tested for % accuracy in reading all items on the dashboard | On average, the % accuracy must be at least 90% | Physicians are tested for % accuracy in reading all items on the dashboard | On average, the % accuracy must be at least 98% |
| INCORRECT | Physician critical care dashboards need to be legible | … | Physicians are tested for % accuracy in reading all the items on the dashboard | On average, the % accuracy must be at least 90% | Physicians are given additional training in dashboard literacy | On average, the % accuracy must be at least 98% |
