Learning object evaluation instrument (LOBE)

  1. What is LOBE instrument ?

    • LOBE is a 4-scale rubric to evaluate the teaching-learning effectiveness of existing e-learning content i.e. learning objects.
    • A learning object is a multimedia teaching-learning resource that contains dynamic visualizations with voice-over (i.e. animations/simulations/videos) as its core component and either all or a subset of the following components :
      • dynamic visualizations with voice-over (i.e. animations/simulations/videos).
      • activities.
      • set of assessment questions.
      • solved example.
      • stepwise lesson plans for teachers.
    • LOBE measures the quality of the learning objects along four dimensions - Content quality (7 questions), Pedagogical alignment (16 questions), Design efficacy (10 questions) and Technology integration (5 questions).The output from the rubric are questionwise scores under each dimension, for a particular learning object.

  2. What is the structure of LOBE ?


    Figure 1: Structure of LOBE ...
    1. What quality dimensions are measured by LOBE ?

      The four quality dimensions measured by LOBE are:

      1. Content quality (C) (6 questions)
      2. Pedagogical alignment (P) (16 questions)
      3. Design efficacy (D) (10 questions)
      4. Technology integration (T) (8 questions) Quality dimensions measured by LOBE

      Quality dimensions measured by LOBE Description of dimension
      1.Content Quality (C) Measures accuracy of content and assessment questions, how well these components connect to real-world scenarios, how updated the content is with recent advances in the topic knowledge etc.
      2.Pedagogical Alignment (P) Measures the extent to which the learning object is learner-centred and promotes higher-order thinking among the learners.
      3.Design efficacy (D) Measures the extent to which the learning object incorporates Multimedia design principles as well as Interaction design principles to promote meaningful learning among the learners.
      4.Technology Integration (T) Measures how well the learning object adheres to graphic design principles and how well the different components of the learning object have been integrated to create a meaningful teaching-learning experience.

    2. What are the features of the questions in LOBE ?
      1. Questions in LOBE are organized along the three dimensions.
      2. Each question in LOBE rubric :
        • Measures a particular aspect of the specific dimension.
        • Specifies which components of the learning object need to be considered for evaluating this question.
        • Is accompanied by explanations on why this question is important for quality evaluation.
    3. Read more

  3. What is the scoring mechanism in LOBE ?

    Each question can be scored along a 4-point scale : Score 0 (Missing); Score 1 (Inadequate); Score 2 (Almost) and Score 3 (Target) Score

    Score Means the quality aspect addressed by the question
    Score 0 (Missing) Is completely missing in the learning object
    Score 1 (Inadequate) Has been inadequately implemented in the learning object, needs lot of improvement
    Score 2 (Almost) Has almost been adequately implemented but there is still scope of improvement
    Score 3 (Target) Has been implemented in the learning object in the ideal way i.e. as recommended by educational technology research
    1. The quality level corresponding to each score is described in detail by the descriptors.
    2. Descriptors are often accompanied by examples. These examples are meant to further assist the evaluator in deciding the level of implementation quality for that question.

  4. What is the scope of LOBE ?

    LOBE is scoped to learning objects which have dynamic visualizations (simulations/animations/videos) as their core component. Also,

    1. LOBE evaluates learning objects after they have been created.
    2. The granularity level of a learning object is taken as a single topic from a textbook chapter, that typically contains 4 and 6 topics
    3. LOBE is currently designed for topics in STEM domains
    4. It has currently been tested for various topics at primary, secondary and high school levels. It may be applicable for college-level STEM topics but has not been tested yet.
    5. LOBE does not evaluate criterion like 'metadata retrieval'.

Target Users

  1. Who can use LOBE ?

    • Quality control team of e-learning companies to evaluate the quality of learning objects produced.
    • Subject matter experts, instructional designers and graphic designers to give meaningful feedback on quality of learning objects.
    • Educators (i.e. Teachers, principals and HODs) to select good-quality learning objects.

  2. When to use LOBE ?

    If you are an e-learning company, you can use LOBE when you want to :

    1. Find the quality assessment scores for your learning objects in terms of Content quality, Pedagogical alignment, Design efficacy and Technology integration
    2. Do strength-weakness analysis for a set of learning objects i.e.
      • Identify the weak spots along each of the quality dimensions (Content quality, Pedagogical alignment, Design efficacy, Technology integration) in your learning objects that need to be addressed to improve quality.
      • Identify the strong points of your learning objects within each of the dimensions.

    If you are an educator (HOD/Principal of a school), you can use LOBE when you want to :

    • Compare between multiple learning objects from different repositories.
    • Evaluate the quality of learning objects from a single repository.

Download LOBE

Click on the list below to download the version of LOBE you want to use.

Name Image About Version Download Download Count
LOBE_LITE
  • LOBE_LITE is an abridged version of LOBE_PREMIUM consisting of 09-10 questions.
  • It is a 4-scale (Missing, Inadequate, Almost, Target) theory-informed checklist for quality evaluation of learning objects.
  • Use LOBE_LITE when you want to do a quick evaluation of the quality of the selected learning object.
  • The robustness of the instrument cannot be vouched for. For thorough evaluation, use LOBE_PREMIUM (38 questions) that has been tested for validity & reliability.
 User Manual
Rubric
LOBE_PREMIUM
  • LOBE_PREMIUM is a 38-question rubric covering four dimensions : Content quality (C) (5 questions), Pedagogical alignment (P) (18 questions), Design efficacy (10 questions) and Technology integration (T) (5 questions).
  • It is a 4-scale (Missing, Inadequate, Almost, Target) rubric that you can use for a thorough evaluation.
  • Robustness of the rubric has been tested with educational technology experts.
 User Manual
Rubric

User Manual


How to use LOBE ?

LOBE can be used in a stepwise manner to score the quality dimensions of a learning object and obtain the quality assessment for a set of learning objects created. The guiding steps are given below.

  • Step 1 - Go through all the different components of the learning object and then answer the questions in LOBE.
    1. You may have to revisit specific components multiple times while responding to the questions in LOBE.

  • Step 2 - Give a score for each question based on the extent to which the level of implementation in the learning object matches the descriptors given for that question in LOBE.
    1. Certain questions can be marked 'Not Applicable' (NA), if :
      • the learning object does not have a certain component.
      • it can be justified that the component is not needed for that context (like lesson plans for student self-learning learning objects).
    2. However, validity of this modified version cannot be vouched for.

  • Step 3 - Compute average percentage scores for the three dimensions of content quality, Pedagogical alignment & Technology integration for a set of at least 30 learning objects.
    1. This will give you an overall picture of how well the learning objects, on an average, fare along the three dimensions on a quality scale of 0 -100.

  • Step 4 - Do a further strength-weakness analysis for the set of learning objects.
    1. This will help locate which specific quality aspects within each dimension needs to be looked into to improve quality
    2. Locate the quality aspects where the learning objects have reached target level they are strong in
    3. This can be done by computing a matrix of percentage of learning objects scoring 0,1,2,3 (in columns) for each question in LOBE (in rows)

Apart from these, LOBE can be used to assess the quality of a specific learning object along the 4 dimensions

Development Process


How was LOBE developed ?

LOBE was iteratively developed in three stages. Refer to Figure 2 below.

Figure 2: Development process of LOBE
  1. Stage 1 - The dimensions & their aspects to be evaluated emerged from literature analysis of existing e-learning evaluation frameworks and instruments
  2. Stage 2 -
    • Questions were framed to measure each of the identified quality aspects.
    • Different levels of implementation for each question were described by writing the descriptors.
  3. Stage 3 - Finally pilot testing of the validity of LOBE was done with:
    • Educational technology experts and
    • Target users of LOBE i.e. quality control team from an e-learning company
  4. LOBE does not evaluate criterion like 'metadata retrieval' since ease of search of open-source resources does not tally with the objective of the learning objects being evaluated

Robustness Testing


How robust is LOBE ?

LOBE has been pilot tested for different forms of validity. This implies LOBE has been tested with educational technology experts, who used existing learning objects, to test if LOBE accurately measures what it intends to measure (validity) i.e. content quality, quality of pedagogical alignment & technology integration in learning objects.

  • LOBE was tested for four types of validity (refer image below) :
Figure 3: Validity testing for LOBE
Visitors Count +10K Since 2018