Genie Testing: Hollywood Studios Double-Head-to-Head

Yesterday (October 23rd) was a gorgeous Saturday in Hollywood Studios, and it was the last day of our marathon Genie testing plan – we’ve all earned a few days of rest … before we do even more testing! And this was our busiest day of testing yet – we had four different testers in Hollywood Studios, attempting to “do the whole park” before they left for the day. And we had two live streams (on YouTube and Instagram) to talk about what was happening in the afternoon. So what happened and what did we learn? Strap in – there’s a lot to cover!

Test 1: Knowledge vs Money

When we ran a Genie/TouringPlans head to head in the Magic Kingdom on Thursday, several people commented that we messed with the results by kicking off after rope drop, because all savvy TouringPlans users know that rope drop is the key to saving time. And while I don’t think all of the members of our community do rope drop, I agree that it’s a key time-saving strategy. So our first test of the day was to pit a savvy TouringPlans tester with a touring plan and early theme park entry against a non-touring plan, late arrival tester equipped with all of the money she needed to pay her way around lines. So what did their days look like?

Tester 1: Early Entry and a Touring Plan

Tester 1 entered the park at 7:45 – well before the advertised early theme park entry time of 8:30. She was able to walk almost all of the way to the entrance of the plaza for Rock’n’Roller Coaster and line up with a cast member to be the first guests on that ride for the day. She used her Touring Plan to choose that step, and to re-optimize after every step for the entire rest of the day. Here was how her day progressed:

• Rock’n’Roller Coaster – 0 minute wait at 8:36 (CMs had everyone skip the preshow)
• Tower of Terror – 1 minute wait at 8:41 (CMs again had everyone skip the preshow)
• Toy Story Mania! – 1 minute wait at 8:58 (this was still 2 minutes before the park opened to offsite guests)
• Millennium Falcon: Smugglers Run (Single Rider) – 1 minute wait at 9:07
• Alien Swirling Saucers – 6 minute wait at 9:18
• Rise of the Resistance – 79 minute wait at 9:33 (this included a 24-minute downtime for the ride)
• Slinky Dog Dash – 62 minute wait at 11:06
• Lunch at PizzeRizzo (our tester’s favorite spot in Hollywood Studios)
• Walt Disney Presents – 25 minutes of relaxed touring
• Runaway Railway – 44 minute wait at 1:13

Our tester joined us for streaming and then called it a day, but she could have seen Beauty and the Beast at 2, walked on to MuppetVision and Star Tours (which were walk-ons at that point in the afternoon) and watched the Frozen Sing-Along Celebration at 3:30. She would’ve been done with her day by 4.

Our tester commented that this was a perfect and “golden” day. She admitted that she was tempted to follow her instincts rather than the plan several times. But if she had, for example, gotten in line for Rise of the Resistance instead of Rock’n’Roller Coaster first thing, that probably would’ve been the only attraction she did before the park officially opened, and then she would’ve lost her advantage over the rest of the crowds.

Early Theme Park Entry was absolutely key here. She finished three attractions before other guests could even get in the park. If she started 30 minutes later, she would’ve had a 20 minute wait for Toy Story Mania at 9:30 instead of 1 minute at 8:58. But it was the combination of early entry and informed strategy that allowed her to be successful and have such a fantastic day. If this test helped you recognize the value of a resort stay, talk to the TouringPlans Travel Agents – they’re top-notch and will get you set up.

Tester 2: Arrive Late and Buy All of the Things

Tester 2 was playing someone that didn’t want to get to the park before 11 am. Maybe that person has kids that don’t want to get up early, or needs a slower start in the morning. That being said, she still did wake up early to purchase Genie+ and make her first return time reservation. She ended up picking Slinky Dog Dash. Since she didn’t log in right at 7, the first available return time was 3 pm – 4 pm. Then, because she wasn’t a resort guest, she had to wait until 9 am to book the Individual Lightning Lane Access return times that she needed. We’ll look at those and the rest of her schedule below:

• Grabbed Slinky Dog Dash return time (3:00-4:00) at 7:00
• Grabbed Rise of the Resistance return time (5:20-6:20) at 9:00
• Grabbed Runaway Railway return time (12:10-1:10) at 9:00
• Grabbed Millennium Falcon return time (4:30-5:30) at 11:04
• Entered the park a little after 11:00
• Frozen Sing-Along Celebration at 11:30
• Runaway Railway via Lightning Lane (LL) – 10 minute wait at 12:10
• MuppetVision – 11 minute wait at 12:31
• Star Tours – 1 minute wait at 1:02
• Grabbed Tower of Terror return time (2:50-3:50) at 1:04
• Walt Disney Presents – 25 minutes of relaxed touring
• Beauty and the Beast at 2:00
• Tower of Terror via LL – 10 minute wait at 2:50
• Grabbed Rock’n’Roller Coaster return time (3:20-4:20) while in line for Tower of Terror
• Slinky Dog Dash via LL – 5 minute wait at 3:05
• Grabbed Toy Story Mania return time (4:35-5:35) while in line for Slinky Dog Dash
• Rock’n’Roller Coaster via LL – 11 minute wait at 3:27
• Grabbed Alien Swirling Saucer return time (3:50-4:50) while in line for Rock’n’Roller Coaster
• Alien Swirling Saucers via LL – 3 minute wait at 4:19
• Millennium Falcon: Smuggler’s Run via LL – 9 minute wait at 4:29
• Toy Story Mania via LL – 6 minute wait at 4:50
• Rise of the Resistance – 13 minute wait at 5:18

Wow. That’s a lot of steps because of all of the return time finding/reserving/using. But it was successful! The pay-your-way tester was able to complete virtually everything in the park between the hours of 11 am and 5:30 pm, typically the most crowded hours of the day. She banked return times for mid- and late-afternoon while doing low-wait or no-wait shows or attractions early in her day. That was HUGE. I’m willing to bet that it’s possible to grab return times all morning while doing another park, and then hop to Hollywood Studios at 2, burn all of the return times and do everything in the park after hopping.

That being said, our tester said it was mentally exhausting trying to keep track of all of the return times. Genie doesn’t push any notifications when your return windows open (or are about to open or about to close). That mental exhaustion, if you do keep up with it, is counter-balanced by the emotional rush of flying past everyone in long standby lines.

Conclusions of this test

In this case, knowledge and money just about balanced out. Both of our testers felt like they had really great days. One got to not rope drop, and one got to not spend additional money. But knowledge vs money may be a bit of a misnomer. Our pay-your-way tester wasn’t flying blind and just randomly grabbing whatever return times looked good. She knew which ones to get first before they booked up, and which to wait on. She knew when she’d be eligible to make another reservation. And she knew to visit the low- and no-wait attractions even before she booked return times for other longer-wait attractions, knowing she’d be able to get them later and it was best to spend peak hours on less popular attractions. So even pay-your-way requires some research or inside knowledge.

Test 2: Does Genie Care About My Preferences?

One of the key claims of the Genie plan for your day is that it takes your preferences into consideration and builds the best day based on what you tell it. Cool? Very cool. In theory. In Thursday’s test, all preferences were almost wholly ignored. Zero fun. So we wanted to test what two people with different inputs would experience as their output. They weren’t allowed to “work ahead” – they had to wait for the times that Genie suggested, because we wanted to mimic what inexperienced park visitors might do. And they weren’t allowed to “hack” Genie with any of the tricks (like mass step deletion to force optimization) we had learned in previous days.

Tester 3: Star Wars Fanatic

Our tester pretended that she only cared about Star Wars things. She picked Star Wars attractions as her must-dos, and noted her interest in Star Wars as a category. And her plan pretty much reflected her preferences! It was a minor miracle! I’ll list her initial plan here, with some breaks for notes:

• Step 1: Walt Disney Presents at 9:10
  – Seriously? Literally anything else would’ve been a better “rope drop” suggestion. Don’t do Walt Disney Presents first. Ever.
• Step 2: Toy Story Mania at 10:10 – 37 minute wait
  – Not a bad suggested attraction, but it’s over an hour after park opening. No need for that much downtime. And it resulted in a much longer wait than if she had gone earlier.
• Step 3: Lunch at Ronto Roasters at 11:15 (more about this step later)
• Step 4: Rise of the Resistance at 12:20 – 93 minute wait
  – This left an hour for lunch, which is potentially not unreasonable. But it was a long wait for Rise.
• Step 5: Droid Depot at 3:10
  – Wait, WHAT?! Disney is upselling her to buy a $100+ droid? Okay, I mean, she selected all of the Star Wars things, but still … bold move. So we assumed she wouldn’t fall for that, and allowed her to attempt a Swaperoo!!!!!

  

• Step 5 (swaperoo’d): Alien Swirling Saucers at 3:10 – 29 minute wait
  – This was the longest wait for Alien Swirling Saucers out of any of the four testers
• Step 6: Millennium Falcon: Smugglers Run at 3:50 – 58 minutes
  – Ouch, another long wait. In fact, the longest wait (by 18 minutes) out of any of the four testers

At that point, our tester bailed for the day. She experienced 5 attractions between 9 am and 5 pm. And two of them were really long waits. If she had finished her plan, it would’ve looked something like this:

• Step 7: Dinner at Docking Bay 7 at 5:30
• MuppetVision at 6:35 (would’ve been a walk on)
• Star Tours at 7:20 (would’ve been a walk on)
• Tower of Terror at 8:10 (would’ve been a very low wait or walk on)

So at the end of the day, no Runaway Railway, no Slinky Dog Dash, no repeated Star Wars attractions that might have made her happy. And definitely not ideal waits. Things were spread out in a way that left odd gaps and breaks throughout the day. Our tester thought it felt awkward and not a pace she would choose for the future.

Tester 4: Blank Slate

The fourth and final tester (or in this case … Testa) followed a Genie plan after giving it no input or preferences. It provided very interesting contrasts to the more aggressive testers in the first matchup, as well as the Star Wars tester.

• Step 1: Toy Story Mania at 9:30 – 20 minute wait
  This might not strike you as odd if you didn’t know that this tester was in the park at 8:30. Genie recognized the park location, but didn’t nothing to fill the first hour. It was painful to watch other guests being efficient with their time as he killed time waiting.
• Step 2: Droid Depot at 10:40
  NO. STOP IT. Droid Depot recommendation for a Star Wars fan? I’ll give it a pass. Droid Depot recommendation for someone with no thoughts and preferences? No. Droid Depot as the second step of an entire day for that person? Absolutely not. Never. Stop it. Do not pass go. I’m appalled.
• Step 2 (swaperoo’d): Walt Disney Presents at 10:10
  At least the swaperoo filled a little of the gap in time. But the swaperoo options were not great. Now both of our free Genie users were doing Walt Disney Presents and Toy Story Mania as their first two steps.
• Step 3: Lunch at Ronto Roasters at 11:10
  Okay, 2-for-2 on recommendations for early lunch at Ronto Roasters. And it was very apparent that our testers weren’t the only ones that got this recommendation. Even though the park wasn’t packed, and 11:10 is a pretty early lunch, every single seat (indoors and outdoors) at Ronto Roasters was taken. Our tester ate lunch on a trash can. But he at least said that the veggie wrap was good. So that’s something.
• Step 4: Millennium Falcon Smugglers Run at 12:10 – 40 minute wait
  This wasn’t a terrible wait, but it was the 3rd attraction of the day and already in the afternoon.
• Step 5: Beauty and the Beast at 1:00
  This step was originally Star Tours at 2 pm at the beginning of the day. But when the tester exited Millennium Falcon, it was swapped out as a “better fit” according to Genie.
• Step 6: Slinky Dog Dash at 3:00 – 54 minute wait
  This step was originally Rise of the Resistance at 3:00. When Rise of the Resistance went down (while Tester 1 was in line around 10:00), Genie took it off of the plan due to downtime. It never reappeared. 54 minutes isn’t a terrible wait for Slinky Dog Dash. But it was still not enjoyable for this tester after wasting a lot of time in the morning. Especially since our pay-your-way tester blew  past him in the Lightning Lane while he was in standby.
• Step 7: Alien Swirling Saucers at 4:15 – 10 minute wait
• Step 8: MuppetVision at 5:15 – 8 minute wait
• Step 9: Dinner at ABC Commissary at 6:00
  Not a bad dining recommendation as far as quick service is concerned. Satisfaction has been increasing at this location.
• Step 10: Vacation Fun (short) at 6:50 – 6 minute wait
  Our tester did not think this vacation day was fun.
• Step 11: Tower of Terror at 7:35 – 8 minute wait
  Crowds were really emptying out of the park by this point. Wait times were really low for almost every attraction.
• Step 12: Wonderful World of Animation at 8:45
  Our tester gave up before this point. The day was too disappointing to continue. But he won the endurance award for being in the park longer than any other tester!

This tester was probably the most disappointed and annoyed of any of our testers today. He evaluated the impact of this plan on him as “I can feel my will to live ebbing away”. The only thing longer than this tester’s day in the park is … his hair.

Conclusions

Despite the disappointment and annoyance levels of our Genie testers, we learned some very interesting things! When conditions changed, the Star Wars steps in his plan mysteriously disappeared, while they stayed on the plan for our Star Wars fanatic. It may be our first example of Genie potentially taking a preference into account. If Rise went down, and Disney knew that they would have to accommodate all ILL reservation holders for that time (and the rest of the day) as well as standby waiters, it makes total sense that they wouldn’t suggest anyone go to the ride for the rest of the day, perhaps unless they had already indicated that it was a must-do. But also, Disney pushed the hard upsell of Droid Depot to a non-Star Wars fan. Weird. And the pace of the day was soooo slooooow while still not hitting all (or many) headliners.

Overall Results

Qualitatively, the differences between the two groups of testers were blindingly obvious. Two testers felt great about their choices and how their day went (which is saying something, given that they’ve been going at this for 5 straight days now). Two testers felt let-down and like they wasted a lot of time.

But enough with the qualitative stuff. Let’s look at the numbers. For these summaries, we’ll assume that testers 1 and 3 finished out their day since we can easily pull the data to see what their waits would’ve been.

• Tester 1 (Touring Plan and early entry) experienced 13 attractions in 7.5 hours. She waited in line for 200 minutes (24 of this was downtime for Rise). The average rating of the attractions that she experienced was 3.88.
• Tester 2 (Pay-Your-Way) experienced the same 13 attractions in a little over 6.5 hours (but with prep time for up to 4 hours before she was in the park). She waited in line for 79 minutes. The average rating of the attractions that she experienced was 3.88. She paid a total of $40 (pre-tax) for her line-skipping powers. That was $15 for Genie+, $15 for Rise of the Resistance, and $10 for Runaway Railway.
• Tester 3 (Genie – Star Wars Fanatic) experienced 8 attractions in 11.5 hours. She waited in line for 226 minutes. The average rating of the attractions that she experienced was 4.0.
• Tester 4 (Genie – No Preferences) experienced 10 attractions in 12.5-13 hours (including the wasted first hour of the day). He waited in line for 143 minutes. The average rating of the attractions that he experienced was 3.78.

Tester 3 got to see all of the Star Wars things … but so did Testers 1 and 2, and they had more attractions in their day and less overall time in lines. It’s a rough comparison. And in Genie, there’s currently no way to set an end time (without pretending that you’re park hopping even if you’re not), so some bigger attractions get shoved to the end of the day, which means you’ll miss them if you don’t spend that full 13 hours in the park. And it’s apparent that just like Disney needs to bump up posted wait times to avoid angry guests, they’re doing the same thing with Genie plans. They don’t want you to feel like you’re falling behind, so they’re padding them with TONS of “bonus” time that equal a lot of waiting around.

Like we’ve mentioned before, the big miss here is lack of a feedback loop. You can’t mark steps complete, so Genie will never know if you’re running ahead (or behind for that matter). And you can’t tell it you really hated its suggestion unless you delete or force a swaperoo, and it doesn’t seem like even those actions are taken into consideration for the rest of the steps in your day.

We had low expectations for Genie today, and we were still disappointed. Genie+ and ILL were worth the purchase price, but Genie plans were a total disaster when compared to savvy strategy.

That wraps up our intense 5-day testing plan. We’ll be back tomorrow with an overall look at things we’ve learned and any tips/tricks we’d recommend based on current conditions. Remember, Disney will keep trying to tweak and update Genie, and we’ll keep testing and trying to stay one jump (or sixteen jumps) ahead. Have any questions or concerns you’d like to share? Leave a comment below!