Emergent space using Rasch pairs analysis/ adaptive comparative judgment

PSEUDO-RANDOM DATA TESTING THE SCALES USED IN RASCH PAIRS ANALYSIS/ ADAPTIVE COMPARATIVE JUDGEMENT

ABSTRACT

Pseudo-random data are used to illustrate the relationship between errors in raw data being comparatively judged and the resulting Rasch pairs location parameters, first for data which are relatively homogeneous and second for data which have various amounts of heterogeneity.  For each data type, various error sizes are used.  Rasch pairs location parameters are demonstrated to be plotted on a contracted scale when the objects appear to be homogeneous.  As space is contracted near concentrations of mass, so is the Rasch scale also contracted when the objects in it are determined to be located very close to one another.

 INTRODUCTION

 There are two interesting feature of Rasch analysis.

The Guttman structure is not a fault with the Rasch model.  The Guttman effect would also cause problems with constructing any physical scale.

  • The Rasch output parameters are on an apparently arbitrary scale, said normally to be +3 to -3, but the scale is often different, e.g. +1 to -1.5. What controls the calculated range?

The Rasch scale is investigated using pseudo random data sets generated for this paper. Different data sets have been generated to have different amounts of true error in their locations and the Rasch output parameters were computed for each such data set to find the corresponding ranges in parameter values.

The question of what determines the range of parameters, i.e. the scale of separation of objects.

METHODS AND RESULTS

In this part, sets of data were generated and the DOS BIGSTEPS Rasch pairs analysis was run on each set of data.  (Ref.: http://www.winsteps.com/a/bigsteps.pdf, Example 13.  Bigsteps is a free DOS version of Winsteps.)

Two types of data were generated using an MS Excel spreadsheet.  In both types of data, twelve objects were compared, pair at a time, thus involving 12C2= 66 comparisons.  There had to be an element of randomness in order for the program to run and a random adjustment to the true value of zero was added to each object for each pseudo-judge in each paired comparison.  Therefore 66×2= 132 quasi-random numbers were generated for each set of data.

In the truly homogenous set of data, each of the twelve objects was set to have a true value of zero.  Four sets of data were generated with different limits to the sizes of random numbers: within -2 to +2, within -1 to +1, within -1/2 to +1/2 and within -1/8 to +1/8 added to the true value of zero.  The winner in each paired comparison was deemed to be the larger of the pair of random numbers.

 

Table 1:  Rasch pairs results for truly homogeneous objects

Data set +/- 2 Data set +/- 1 Data set +/- 1/2 Data set +/- 1/8
Object no. Location parameter Object no. Location parameter Object no. Location parameter Object no. Location parameter
  3  0.91   7  0.97   1  0.56   2  0.56
  4  0.52 11  0.97   5  0.56   4  0.56
  8  0.52   1  0.57   7  0.56   5  0.56
  2  0.17 10  0.57 11  0.56 10  0.56
  6  0.17   3  0.19   6  0.20   8  0.20
  9 -0.18   9  0.19   8  0.20   9  0.20
10 -0.18   2 -0.18 10  0.20 11  0.20
11 -0.18   5 -0.18   2 -0.16   1 -0.16
12 -0.18   4 -0.55   3 -0.16   7 -0.16
  1 -0.53   6 -0.55   4 -0.16 12 -0.16
  5 -0.53 12 -0.55   9 -0.94   3 -0.94
  7 -0.53   8 -1.45 12 -1.42   6 -1.42
Range of Location parameters 1.44 2.42 1.98 1.98
Ave. error per object 0.45 0.47 0.46 0.46
S.D. of object errors 0.01 0.03 0.03 0.03

 

Table 2:  Intercorrelations between order of merit of location parameters for truly homogeneous objects

Data set +/ -2 +/ -1 +/ -1/2
+/- 1 -0.490
+/- 1/2 -0.049 0.224
+/- 1/8 -0.154 -0.196 0.420

The correlation coefficients for the orders of merit of objects in the four Rasch analyses vary from -0.49 to +0.42, with median value -0.05.  These low values indicate that a lack of association can be expected when the objects are truly equal but with some random element added at every place in the 66 paired comparisons.

The four ranges of Rasch parameter location values are: 1.44, 2.42, 1.98 and 1.98 with median value 1.98.  (Where, for example, 1.98 = 0.56 –  -1.42 .)

This pattern of Rasch analysis results is what is desirable in a test of comparability of twelve scripts chosen say from two different tests intended to be equivalent or interchangeable.  If replications were undertaken, one would hope for different results each time as a sign of homogeneity of objects.

The second type of data is for truly heterogeneous objects.  True location values of objects were allocated as 1, 2, …, 12,  for the twelve objects.  Adjustments were made to the true values as for the homogeneous data.  Random values were added of six different size limits: within +/-2, within +/-4, within +/-6, within +/-8, within +/-10 and within +/-12.  As the size limit of the random adjustment decreases, the data should look more heterogeneous, i.e. should look less and less like the first data set in Tables 1 and 2.  Five replications of data were made for each size of adjustment (except +/-2). There are too many results to show in as much detail as in Tables 1 and 2, instead summaries of results are shown in Table 3.

 

Table 3   Summary of results for truly heterogeneous objects.

Data set+/- 12

 

Replication

Range of parameters location Ave. error per object S.D. of object errors Correlation with true location(12 objects)
1 1.45  (0.54 to -0.91) 0.45 0.02 0.594
2 2.56  (1.53 to -1.03) 0.50 0.03 0.730
3 4.09  (1.67 to -2.42) 0.54 0.08 0.767
4 2.39  (1.43 to -0.96) 0.47 0.03 0.589
5 3.24  (2.21 to -1.03) 0.50 0.08 0.791
Median 2.56 0.50 0.03 0.730

 

Data set+/- 10

 

Replication

Range of parameters location Ave. error per object S.D. of object errors Correlation with true location
1 2.56  (1.03 to -1.53) 0.50 0.03 0.812   (12)
2 3.00  (1.50 to -1.50) 0.49 0.04 0.756   (12)
3 2.94  (1.47 to -1.47) 0.48 0.04 0.653   (12)
4 2.74  (1.11 to -1.63) 0.52 0.04 0.699   (12)
5 2.98  (0.89 to -2.09) 0.52 0.07 0.709   (11)
Median 2.94 0.50 0.04 0.709   (12)

 

Data set+/- 8

 

Replication

Range of parameters location Ave. error per object S.D. of object errors Correlation with true location
1 8.09  (4.67 to -3.42) 0.72 0.17 0.832   (11)
2 3.14  (1.57 to -1.57) 0.51 0.04 0.905   (12)
3 4.09  (1.67 to -2.42) 0.55 0.08 0.737   (12)
4 3.70  (2.21 to -1.49) 0.55 0.07 0.703   (11)
5 2.53  (1.51 to -1.02) 0.49 0.04 0.777   (12)
Median 3.70 0.55 0.07 0.777   (12)

 

Data set+/- 6

 

Replication

Range of parameters location Ave. error per object S.D. of object errors Correlation with true location
1 4.32  (2.16 to -2.16) 0.61 0.09 0.756   (10)
2 4.72  (1.93 to -2.79) 0.66 0.09 0.828   (11)
3 3.71  (1.86 to -1.85) 0.58 0.03 0.884   (12)
4 5.14  (2.58 to -2.56) 0.58 0.09 0.835   (12)
5 3.51  (1.75 to -1.76) 0.56 0.04 0.870   (12)
Median 4.32 0.58 0.09 0.835   (12)

 

Data set+/- 4

 

Replication

Range of parameters location Ave. error per object S.D. of object errors Correlation with true location
1 5.34  (2.60 to -2.74) 0.66 0.11 0.913   (11)
2 7.24  (3.65 to -3.59) 0.75 0.09 0.958   (12)
3 5.46  (2.73 to -2.73) 0.54 0.08 0.798( 9)
4 5.50  (2.75 to -2.75) 0.68 0.12 0.893   (11)
5 5.79  (2.89 to -2.88) 0.64 0.11 0.873   (12)
Median 5.50 0.66 0.11 0.873   (11)

 

Data set+/- 2

Replication

Range of parameters * location Ave. error per object S.D. of object errors Correlation with true location
1 1.10  (0.55 to -0.55) 0.45 0.02 n.a.    (4)

*Based on only four objects as results for eight objects failed to converge due to a Guttman pattern of data.

Replications were thought to be unnecessary.

 Heterogeneous objects with an inbuilt randomness of +/- 12 are those, in this second type of data, which are most like homogenous objects.  As the size of random adjustment decreases to +/- 2 the data act more like the heterogeneous data that their true unadjusted values are.  By the time the adjustment is only +/- 2, the Rasch pairs analysis cannot cope with eight of the objects and only four objects have converged location parameters.   The best sign of heterogeneous data is a failure of the program to produce results.  This is because much of the data are in a Guttman pattern and a Guttman pattern indicates non-locality of objects.

As the random element gradually decreases, the correlation of location parameters with the true locations tends to increase:  0.730 –>  0.709  –>  0.777 –>   0.835  –>  0.873  (median values in Table 3).  At the same time, the range of location parameters also markedly increases:  2.56  –>  2.94  –>  3.70  –>  4.32  –>  5.50 (median values in Table 3).  Thus, an indicator of the homogeneity of objects is the compressed range of location parameters.  The Rasch analysis assesses that objects are relatively homogeneous and compresses the mark scale for them.  Thus the Rasch analysis is maybe acting rather like the metric of physical space near a concentration of mass.  The average error in the location parameter is calculated by BIGSTEPS and this value gradually rises from 0.50 to 0.66 as the data become more heterogeneous in nature and the location parameter sizes increase.  The standard deviation of the location parameters also rises from 0.03 to 0.11, at the same time.

 

CONCLUSION

This paper shows that a Rasch analysis compresses its location parameter space according to the level of uncertainty in making judgements within that space.  The more uncertain the judgements, the more compressed are the points on the scale.

 

 

References

Andrich, D.   Relationships Between the Thurstone and Rasch Approaches to Item Scaling,  Applied Psychological Measurement,Vol. 2, No. 3, Summer 1978, pp419-460.

Pollitt, A. and Crisp, V.  Could Comparative Judgements of Script Quality Replace Traditional Marking and Improve the Validity of Examination Questions?   Paper presented at the British Educational Research Association Annual Conference, 2004.

 

 

 

20 May 2011

(Version 2.0, 20 May 2011)

(Version 3.0, 30 August 2014)

(Version 3.1, 20 September 2014)

 

 

 

Advertisements
This entry was posted in physics and tagged , , , , , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s