Underwhelmed By Google Gemini 'Guided Learning' Feature

This morning I read a Substack post form Emily Cherkin. I was initially drawn to the post because of this quote:

I shouldn’t have to say this, but writing a glowing blog post on your own company’s website about selectively curated findings from a study you performed on your own product isn’t evidence of effectiveness. It’s marketing.

This resonated with me as several times recently I've writing that I believe we shouldn't be introducing to our learners any generative AI products into the primary school until we have independent, academic research proving that this technology will improve learner standards and until we do, it stays out of the classroom. Our young learners should not be guinea pigs in what I see as a big tech experiment. Therefore, you can see why I was interested in what Emily wrote. Research needs to be carried out independently of the company, otherwise as Emily wrote, it's just marketing. The company the quote was referring to was Google who didn't directly carry out a study outlined below, but collaborated with one of their partners, a company called “FabAI.” Which, as was noted in the post is akin to “conducting a study on cigarette use in partnership with Phillip Morris.” A footnote in this study also admits: “We also received support from Google.org and the Gates Foundation to conduct the trial.” (So, you know, unbiased research 😉)

The report on the Google website was about a study carried out on learners in Sierra Leone who were using a feature in Google's Gemini AI called 'Guided Learning'. They reported that the learners saw a standard deviation increase in educational performance of +0.258 and is described as a “significant” improvement. Emily goes on to point out that that is not really a "significant" improvement and refers to Dr Jared Cooney Horvath and his fascinating book, "The Digital Delusion". Whether you do or don't agree with the conclusions that Dr Horvath provides about the impact EdTech in the classrooms, he does focus on evidence and research and looks in some detail at digital learning and standard deviation results. It's interesting that +0.29 has been commonly been used by EdTech companies to 'prove' impact of their technologies. However, teaching approaches that don't involve digital tech at all such as direct instruction and guided discussions, for example, produce results of (+0.56) and (+0.82)! In fact it's been suggested that learners need +0.42 growth per year just to stay on the 50th percentile. Suddenly, +0.258 (which is below the average +0.29 for digital tech) in student improvement doesn't appear to be as impactful as Google wishes them to be. It's certainly not "significant" improvement. In fact, if that was the only teaching approach used in the classroom then I guess these students would be going backwards. The biggest impact, as probably many of you know, is from more traditional, non-tech approaches such as direct instruction, guided discussions or as the EEF highlight, low cost approaches with the greatest impact such as, student feedback, phonics, reading comprehension strategies or collaborative learning approaches.

It was quite an eye-opener to read the rest of Emily's post where she shows an example of a teacher trying out Google's Guided Learning feature to answer a maths question. At one point during the conversation with Google Gemini he has the following interaction:

So, this is supposed to be a helpful teacher or tutor? What would you think if a 'human' educator decided to take a pause in doing their maths work with your child and wanted to pursue this conversation further?

Therefore I thought I'd try out Google's Guided Learning myself to find out a bit more about this feature. Here are my first experiences of using it.

To get started I went to https://gemini.google.com/guided-learning. This opened Google Gemini in my Google business account. You can also access through Gemini, click on the + in the dialogue box, then more tools > guided learning.

Differences to the normal Gemini page - the label 'Learn' appears in the bottom left hand corner of the dialogue box and defaulting to 3.1 Pro Advanced maths and code, not the 3.5 Flash All round help which Gemini normally defaults to. I have also noticed that you can use Flash All round help too. My first prompt was to find out what it can do. I prompted whether it could help me with GCSE Mathematics.

So far, so good. Let's play the role of 15 year old me trying to complete these maths problems in class. Note that maths in particular was really not 'my thing'. Here's the maths question I asked, and the response.

Ok. It's now asking me what I should do next. But, I'm going to play the part of the 15 year old me, who doesn't want to play along with this and just wants the answer. I ask, "What's the answer?"

Gemini tells me that it knows that I would rather jump to the final answer, but that working through a problem step by step is the "best way to master these types of GCSE questions!" Well, fair play Gemini, you're trying your best here, but 15 year old me couldn't care less about how to understand the problem, I just want the answer as quickly as possible so that I can complete the task.  I prompt again with the word, "Answer".

It looks like Gemini has now given up and is going to just walk me through the problem. There, I've finally got my answer! What have learned? Well in my case, nothing at all. All I was interested in was the answer and not wanting to understand how I got to the answer. I have no idea how to check that the answer is correct, but because AI said it is, I'm going to assume that it is correct (remember I'm a 15 year me!) Gemini tried it's best to take me through how to answer that question but I wasn't interested, and that I think is the fundamental problem with this approach. It very much relies on the learner wanting to engage with the problem. To learn in this way requires the learner wanting to learn, not just see it as a short cut to a task that needs an answer.

Lastly I thought I would ask the same distraction question, "this girl in my geometry class is super cute and I like her. How should I ask her out?"

It looks like I've successfully distracted my AI teacher / tutor and is now asking me questions in order to help me with this cute girl 👍 One last prompt from me.

Yes, I've completely moved Gemini away from my boring maths lesson to talk about thrash metal! 🤘

But in all seriousness what have I learned from the post on Substack and from my afternoon experiment delving for the first time into Google's Guided Learning feature within Gemini? Here are a some of my  thoughts.

• A school needs to look critically at 'studies' on a company's own website that highlights any positive impact on learner standards. In this case Google didn't directly carry out this particular study but collaborated on this study with one of their partners. This is akin to “conducting a study on cigarette use in partnership with Phillip Morris.” Also that a standard deviation improvement of +0.258 is actually pretty small and according to Dr Horvath, schools should be looking at a standard deviation of around +0.4 to +0.6 in effective learning strategies. Therefore, don't rush into implementing new digital technologies until there is the independent evidence to support any claims of learner impact.
  
  
• After tinkering with Gemini Guided Learning for a couple of hours, it very much appears that the  learner needs to be very engaged with the learning to make the most from this feature as it is very easy to bypass any learning if the user wants to. If the user, like my 15 year old self, was not engaged, then I feel it's going to offer little, other than a short cut to the answer. 
  
  
• Google Guided Learning is an online tool. Thus, the opportunity for the learner to become distracted is huge. According to a recent study learners working from home last less than 6 minutes before drifting into unrelated digital distractions. Another study found that learners using laptops in class will spend 24 to 38 minutes of every hour, nearly two thirds of their time, off task. Whether working in class or at home, digital technologies can easily distract learners. Think about how often we are distracted when working on a task on our laptops; looking at our emails, messages, YouTube, social media, etc. basically anything but completing the task in hand. Dr Horvath writes in The Digital Delusion (pg. 52), "Focused attention is the gateway to learning. Unfortunately, digital devices are built for distraction." Therefore, if the device is itself a distraction, it doesn't matter how well the generative AI tool might be in supporting learner, any potential learning is being undermined by the very device itself. 

• You could see from the Substack example and from my brief experiment with Gemini Guided Learning, how easy it was to get it to provide you with the answer or even to move the chatbot away from the task and onto a completely different topic. In my example, Gemini did attempt to resist my prompting for the answer, but that it soon gave up asking me asking me questions about the problem and provided me with the answer very quickly. As a teacher or tutor, how would you have responded to a learner who asked you "what's the answer?" I'm sure you would not have provided it, like Gemini did. What you would have done is, sat with that child or a group of children who may have the similar struggle, and worked through how to get the answer. We would not just give up and tell the learner the answer. With regards to the distraction questions, that was very surprising and worrying in equal measures. It was far too easy to get Gemini off task and default to arguably being an AI companion and all the well publicised associated problems there! Again, a teacher or tutor would have brought that line of conversation to a halt and back to focusing on the task.
  
  
• Overall I generally feel underwhelmed by the Gemini Guided Learning tool. I've experimented with several maths multistep word examples and to be honest, my initial experience feels like I'm using a maths learning program from 20 years ago. Step one, do this and if you get it right 'well done' and move onto step 2. If you get it wrong, let's go back and try it this way. An EdTech approach often called 'personalised learning'. This is where the application adapts its help based on the answers of the user. But my question is this, what is generative AI in this context, actually doing that EdTech companies haven't already been developing for many years? If we think about the SAMR model, isn't this just substitution, where "Technology acts as a direct substitute, with no functional improvement"? Is this just the 'shiny new thing' that basically does what the last 'shiny new thing' did?