istation Indicators of Progress (ISIP)

Reading

Rating Summary

Classification Accuracyhalf bubble
GeneralizabilityModerate Low
Reliabilityfull bubble
Validityfull bubble
Disaggregated Reliability and Validity Datahalf bubble
Efficiency
AdministrationIndividual Group
Administration & Scoring Time13 - 21 Minutes
Scoring KeyComputer Scored
Benchmarks / NormsYes
Cost Technology, Human Resources, and Accommodations for Special Needs Service and Support Purpose and Other Implementation Information Usage and Reporting

Cost per student for K-5: $5.00.

Included in the price are manuals and test materials, directions for administration, test forms, technical manuals, and protocol per student.

Internet access is required for full use of product services.

Testers will require 1-4 hours of training.

Paraprofessionals can administer the test.

Alternate forms available for benchmarking.

istation
800 E. Campbell Road Suite 224
Richardson, TX  75081
Phone: (866) 883-READ

Visit http://istation.com
Tech support: support@istation.com

Training manuals are included and should provide all implementation information.

Ongoing technical support is provided.

ISIP is an engaging computerized adaptive assessment of reading ability, including subtests in phonemic awareness, letter knowledge, alphabetic decoding, spelling, text fluency, vocabulary, and comprehension.

ISIP can be administered simultaneously to an entire classroom in 15-20 minutes, with immediate scoring and results. ISIP includes comprehensive reporting for teachers and parents, as well as downloadable directed learning resources for differentiated instruction. Being adaptive, ISIP automatically adjusts the difficulty of items delivered to limit the amount of frustration or boredom often associated with traditional assessment.

ISIP can be used alone for screening, benchmarking, or progress monitoring. ISIP can also be used in conjunction with istation Reading as a comprehensive tiered instruction and intervention solution.

Ability scores are estimated using Bayesian EAP with an informative prior under a 2PL unidimensional IRT model. Reported scale scores are generated through a linear transformation of the raw IRT-based ability scores. Abilities for each of the subtests (phonemic awareness, letter knowledge, alphabetic decoding, spelling, vocabulary, and comprehension) are estimated separately based on examinee response patterns to the items adaptively administered. An overall reading ability is estimated after all of the appropriate subtests are given based on the responses from all items. Text fluency is not IRT-based, and is calculated based on the number of items correct, the number of items attempted, and the time engaged with the maze passage.

Group administered.

 

Classification Accuracy

Classification Accuracy in Predicting Proficiency on the ITBS Reading Scale (for 1st & 2nd grade) and TAKS Reading Scale (for 3rd grade)


 
1st & 2nd Grade
n=143
3rd grade
n=77
False Positive Rate 0.034 0.014
False Negative Rate 0.167 0.142
Sensitivity 0.833 0.857
Specificity 0.966 0.986
Positive Predictive Power 0.833 0.857
Negative Predictive Power 0.966 0.986
Overall Classification Rate 0.944 0.974
AUC (ROC) 0.888 0.864
Base Rate 0.168 0.091
Cut Point: -0.035 0.369

 

Generalizability

Description of study sample:

  • Number of States: 1
  • Size: 416
  • Gender:
    • 51.7% Male
    • 48.3% Female  
  • SES: 55.8% eligible for free or reduced-price lunch
  • Race/Ethnicity:
    • 44% White, Non-Hispanic
    • 18.3% Black, Non-Hispanic
    • 31% Hispanic
    • 6.3% Asian/Pacific Islander
    • 0.5% Other
  • Disability Classification:
    • 3.6% Speech/Language
    • 0.7% LD
    • 0% MR
    • 0.2% Other
  • First Language:
    • 18.0% qualify for ESL services
    • 16.1% received ESL services
  • Language Proficiency Status: 7.7% in a bilingual classroom

Reliability

Type of Reliability Age or Grade n (range) Coefficient SEM
Range Median
Alpha K – 3rd 73 – 102 0.972 – 0.983 0.982 ≤ 0.314
Test-retest K – 3rd 76 – 96 0.836 – 0.934 0.900 ≤ 0.314

 

Validity

Type of Validity Age or Grade Test or Criterion n (range) Coefficient
Range Median
Predictive 1st & 2nd ITBS Reading 137-143 0.845-0.895 0.886
Predictive 3rd TAKS Reading 64-77 0.695-0.741 0.719
Construct K-2nd CTOPP Sound Matching 223 0.662  
Construct K-2nd CTOPP Segment. Words 223 0.620  
Construct K-2nd CTOPP Blending Words 220 0.702  
Construct K-2nd CTOPP Blending Nonwords 220 0.650  
Construct K-1st WLPB-R Letter Word Identification 120 0.711  
Construct K-2nd WIAT Target Words 193 0.589  
Construct K-2nd WLPB-R Word Attack 316 0.830  
Construct K-2nd TOWRE Phonemic Decoding 313 0.838  
Construct K-2nd TOWRE Sight Words Efficiency 313 0.811  
Construct 1st-3rd WIAT Spelling 288 0.875  
Construct 1st-3rd WJ-III Spelling 293 0.890  
Construct K-3rd PPVT 411 0.814  
Construct K-3rd WLPB-R Synonyms 414 0.836  
Construct 1st-3rd WIAT Reading Comprehension 288 0.682  
Construct 1st-3rd WLPB-R Comprehension 287 0.794  
Construct 1st-3rd GORT Comprehension 291 0.621  

Content Validity:
Evidence of content validity was established through a series of steps. First, early reading content experts Drs. Patricia Mathes and Joe Torgesen created ISIP assessment items in the five areas of reading: phonemic awareness, letter knowledge, alphabetic decoding, spelling, vocabulary, and comprehension. Next, the items underwent expert review and revision. Then, the items were operationally used in a previous version of ISIP and revised as necessary. For the new ISIP, the items were calibrated under a 2PL IRT model. Finally, item parameters were examined and those items with unacceptable fit statistics with regards to the subtest to which they belonged were removed from the pool. Being an IRT-based test, CVR or Cohen’s K were not calculated for the 1,600 items. However, based on the process used to establish content validity, the items in the operational pool grouped by subtest are believed to be accurate representations of the domain in which they intend to measure. Construct validity evidence seem to support this statement.

Disaggregated Reliability, Validity, and Classification Data for Diverse Populations

Disaggregated Reliability

Type of Reliability

Age or Grade

n (range)

Coefficient Range

Coefficient Median

SEM

Information (including normative data)/Subjects

alpha

K-3rd

12-21

0.968-0.990

0.985

≤ 0.314

Students who qualified for ESL services. Seven administrations from Oct 2008 to Mar 2009. SEM used as a stopping criterion for adaptive testing.

alpha

K-3rd

52-89

0.975-0.981

0.978

≤ 0.314

Students who did not qualify for ESL services. Seven administrations from Oct 2008 to Mar 2009. SEM used as a stopping criterion for adaptive testing.

alpha

K-3rd

5-20

0.939-0.983

0.972

≤ 0.314

African American students. Seven administrations from Oct 2008 to Mar 2009. SEM used as a stopping criterion for adaptive testing.

alpha

K-3rd

6-45

0.972-0.995

0.976

≤ 0.314

Hispanic students. Seven administrations from Oct 2008 to Mar 2009. SEM used as a stopping criterion for adaptive testing.

alpha

K-3rd

27-59

0.975-0.987

0.979

≤ 0.314

Caucasian students. Seven administrations from Oct 2008 to Mar 2009. SEM used as a stopping criterion for adaptive testing.

alpha

K-3rd

3-10

0.954-0.992

0.983

≤ 0.314

Asian students. Seven administrations from Oct 2008 to Mar 2009. SEM used as a stopping criterion for adaptive testing.

 

Disaggregated Validity

Type of Validity

Age or Grade

Test or Criterion

n (range)

Coefficient Range

Coefficient Median

Information (including normative data)/Subjects

Predictive

1st & 2nd

ITBS Reading

31

0.812-0.883

0.869

African American students. ISIP administered Fall 2008. ITBS administered Spring 2009.

Predictive

1st & 2nd

ITBS Reading

44-45

0.805-0.917

0.883

Hispanic students. ISIP administered Fall 2008. ITBS administered Spring 2009.

Predictive

1st & 2nd

ITBS Reading

53-58

0.841-0.886

0.854

Caucasian students. ISIP administered Fall 2008. ITBS administered Spring 2009.

Predictive

1st & 2nd

ITBS Reading

7

0.762-0.931

0.884

Asian students. ISIP administered Fall 2008. ITBS administered Spring 2009.

Predictive

3rd

TAKS Reading

5-10

0.405-0.843

0.636

African American students. ISIP administered Fall 2008. TAKS administered Spring 2009.

Predictive

3rd

TAKS Reading

6-13

0.553-0.849

0.701

Hispanic students. ISIP administered Fall 2008. TAKS administered Spring 2009.

Predictive

3rd

TAKS Reading

51-53

0.631-0.738

0.701

Caucasian students. ISIP administered Fall 2008. TAKS administered Spring 2009.