PALS

Reading (Gr. 1-3)

Rating Summary

Classification Accuracyfull bubble
GeneralizabilityModerate High
Reliabilityfull bubble
Validityfull bubble
Disaggregated Reliability and Validity Datahalf bubble
Efficiency
AdministrationIndividual Group
Administration & Scoring Time23 - 43 Minutes
Scoring KeyComputer Scored
Benchmarks / NormsYes
Cost Technology, Human Resources, and Accommodations for Special Needs Service and Support  Purpose and Other Implementation Information Usage and Reporting

Cost per student for 1-3: $3.80 (materials only); $6.04 materials plus Online Score Entry and Reporting System  (based on a class of 25 students).

PALS 1-3 Teacher Set includes everything a new teacher needs to screen 25 students in fall and spring and an Assessment Training CD. Includes the following: an Administration and Scoring Guide, fall and spring Class Summary Sheets, a Student Packet, a Teacher Packet, fall and spring Student Spelling Sheets (masters), and 25 fall and spring Student Summary Sheets.

$6.10 per Online PALS Teacher Account.

$2.00 per Online PALS Student Account.        

Internet access is required for full use of product services.

Testers will require less than 1 hour of training.

Developer strongly recommends that teachers screen their own students, though others (e.g., paraprofessionals) may as well administer the test.

PALS Marketplace
PO BOX 2616
Charlottesville, VA 22902

Phone: 1-888-924-7257
Fax: 1-888-924-7258

E-mail:
order@palsmarketplace.com

Website: http://www.palsmarketplace.com/

Training manuals should provide all implementation information.

The Phonological Awareness Literacy Screening (PALS) is a criterion-referenced screening, diagnostic, and progress monitoring tool.  PALS consists of two instruments (PALS-K and PALS 1-3) that measure young children’s knowledge of important literacy fundamentals:  phonological awareness, alphabet awareness, letter sound knowledge, spelling, concept of word, word recognition in isolation, and oral passage reading. The major purpose of PALS is to identify students who may be at risk for reading difficulties and who need additional reading instruction beyond what is provided to typically developing readers. The second purpose of PALS is to provide teachers with explicit diagnostic information about what students know and need to know about the fundamental components of literacy that may be used to target instruction to meet students’ needs. The third purpose of PALS is to monitor students’ progress and determine the effectiveness of instruction or intervention.

PALS 1-3 uses a gated three-tiered approach that differentiates the administration of the assessment based on students’ scores. The first tier (or Entry Level) measures a student’s general level of word knowledge measured by a word recognition and spelling task. If a student does not meet the Entry Level benchmark for his or her grade level further assessment is required. Level A assesses the accuracy, fluency, rate, and comprehension of a student’s oral reading in context.  Level B assesses emergent and beginning reading essentials in alphabetic knowledge and concept of word. If Level B benchmarks are not met, students are routed to Level C for a more in-depth evaluation of phonemic awareness skills, including blending and segmenting speech sounds.

Raw scores from each PALS 1-3 Entry Level task are summed to produce the Entry Level Summed Score; this consists of Word Recognition in Isolation at the appropriate grade level (20 points max), and Spelling (maximum varies across grades 1, 2, and 3). To compute AUC using ROC analysis, we converted Entry Level Summed Scores to simple standard scores (z-scores).

Group/Individually administered.

 

Classification Accuracy

  Predicting performance on the Stanford-9 Achievement Test (Reading)
1st Grade
n=1,088
Predicting Risk on Spring 3rd Grade Standards of Learning (SOL) Reading
3rd Grade
n=500
False Positive Rate 0.298 0.092
False Negative Rate 0.078 0.592
Sensitivity 0.922 0.408
Specificity 0.702 0.908
Positive Predictive Power 0.336 0.802
Negative Predictive Power 0.982 0.628
Overall Classification Rate 0.733 0.670
AUC (ROC) 0.922 0.790
Base Rate 0.141 0.476
Cut Point: The 23rd percentile on the Stanford-9 Achievement Test (Reading) was used as the cut score as it represents the top band in the 3rd stanine, which can be used as a gross indicator of relative risk. Virginia’s SOL for Reading results in scaled scores ranging from 0 – 600, with 400 representing pass, or ‘proficient’ (used in this classification analysis), while a score of 500 represents ‘advanced pass’.

 

Generalizability

Description of study sample:

  • Number of States: 1
  • Size: 1,088
  • Gender:
    • 52.7% Male
    • 47.3% Female  
  • Race/Ethnicity:
    • 50.1% White, Non-Hispanic
    • 45.5% Black, Non-Hispanic
    • 1.0% Hispanic
    • 0.1% American Indian/Alaska Native
    • 1.6% Asian/Pacific Islander
    • 1.2% Other
  • Disability classification:
    • 5.2% speech/language impairment
    • 1.2% learning disability
  • Language proficiency status: 2.1% received ESL services

 

Cross Validation Study Description of study sample:

  • Number of States: 1 (Virginia)
  • Size: 63,626
  • Gender:
    • 51.0%  Male
    • 49.0%  Female
  • SES: 37.2% (median) eligible for free or reduced-price lunch
  • Race/Ethnicity:
    • 56.8% White, Non-Hispanic
    • 26.9% Black, Non-Hispanic
    • 8.8% Hispanic
    • 0.3% American Indian/Alaska Native
    • 4.1% Asian/Pacific Islander
    • 3.1% Other
  • Disability Classification
    • 5.0% Speech/Language Impairments
    • 2.3% Learning Disability
    • 1.8% Developmental Delay
    • 1.3% Other Health Impairment
    • <1.0% Other Disability
  • Language proficiency status: 6.0% received ESL services

Reliability

Type of Reliability Age or Grade n (range) Coefficient
Range Median
Cronbach’s alpha: entry level tasks 1 6,525 – 14,741 0.66 – 0.96 0.81
Cronbach’s alpha: Word recognition in isolation K – 3 25 – 617 0.81 – 0.96 0.91
Cronbach’s alpha: Word recognition in isolation K – 3 4,668, 4,541, and 4,387 0.92 – 0.93 0.92
Cronbach’s alpha: Spelling K – 3   0.86 – 0.92 0.90
Inter-rater: Word Recognition K – 3 45 – 63 0.98 – 0.99 0.98
Inter-rater: Spelling K – 3 130 – 375 0.99 – 0.99 0.99
Inter-rater: Oral Reading in Context K – 3 478 0.94 – 0.98 0.96

 

Validity

Type of Validity Grade Test or Criterion n (range) Coefficient
Range
Predictive Grade 1, 2 Stanford-9 Total Reading 739 – 766 0.73, 0.63
Predictive Grade 3 Virginia SOL 277 0.60
Concurrent Grade 1 Qualitative Reading Inventory (QRI-II) 146 0.73
Concurrent Grades 1 – 3 QRI-II 679 0.73-0.90
Concurrent Grades 1 – 3 Developmental Reading Assessment (DRA 179 (instructional) 96 (independent) 0.82 0.81
Concurrent Grade 1 California Achievement Test (CAT/5):Total Reading Scaled Score 195 0.75 (entry level) 0.70 (spelling)
Predictive Grade 1 Fall 2007 2nd grade Entry Level Summed Score   0.792
Predictive Grade 1 Fall 2007 2nd grade Entry Level Summed Score   0.822
Predictive Grade 2 Fall 2007 3rd grade Entry Level Summed Score   0.810
Predictive Grade 2 Fall 2007 3rd grade Entry Level Summed Score   0.846
Construct   As one piece of evidence in support of the construct validity of PALS 1-3, we conduct Principal Component Analyses on the tasks that make up the PALS 1-3 Entry Level Summed Score each year, based on statewide data.  These analyses consistently yield a single factor (eigenvalue greater than 1.0) that accounts for more than three quarters of the variance in these scores.
Content   The content validity is supported by the process of item development (described in the Technical Reference, Section III, pp. 11 – 19).  The guiding principles underlying this process were that (a) tasks and items were a representative sample of tasks from other early literacy screening instruments, (b) items had a history of use in phonological awareness and early literacy research, and (c) items were aligned with Virginia’s Standards of Learning for English (Reading).  Each of these principles was further supported by the process of review by an Advisory Panel of literacy professionals from across Virginia, and by an External Review Panel, which consisted of nationally recognized experts in the field of reading, communication sciences, or psychology.

 

Disaggregated Reliability, Validity, and Classification Data for Diverse Populations

Classification Accuracy in Predicting Proficiency on the Stanford 9: By Race/Ethnicity
  Black
n = 498
White
n=546
False Positive Rate 0.367 0.235
False Negative Rate 0.054 0.132
Sensitivity 0.946 0.868
Specificity 0.633 0.765
Positive Predictive Power 0.369 0.284
Negative Predictive Power 0.981 0.982
Overall Classification Rate 0.691 0.774
AUC (ROC) 0.903 0.926
Base Rate 0.185 0.097
Cut Points 23rd percentile 23rd percentile

Disaggregated Reliability

Type of Reliability   Grade Coefficient
Range Median
Summed Score Reliability (internal consistency) Across 4 levels (quartiles) of school-level SES K, 1 0.73 – 0.85 0.78
Summed Score Reliability (internal consistency) Across race/ethnicity groups K, 1 0.73 – 0.88 0.82