Have you ever watched motorsports on TV, and struggled to conceptualize what the commentators are talking about? Swooping camera angles and the limited vision they provide can make it difficult to understand how Turn 4 relates to Turn 7, or where they even are in relation to the chicane complex just past Turn 9. There could be a solution to this, wherein virtual reality devices display the on-track action from an all-encompassing, birds-eye view.
John LePore is an “Emmy-Nominated Futurist and Creative Leader,” in his own words, and now runs a new consultancy called Black Box Infinite. He’s worked heavily in the visual effects industry, with credits in over 25 films in the Marvel Cinematic Universe alone, and also specializes in user experience projects. He’s a big-time car enthusiast and obsessed with Formula 1. That led him to develop an idea for a killer app for the Apple Vision Pro—one that would make the most of the mixed-reality headset and its unique capabilities. If you’re not familiar with mixed reality, it’s like a VR headset that overlays 3D graphics over your view of the real world.
“My father’s a racing instructor and I’m always eager to move forward tech in automotive & motorsports,” LePore told us. “I’ve designed emerging experiences (both fictional & practical) for most of my career. Going back and forth between creating gadgets for sci-fi films and creating an instrument cluster in a car (designed to keep your eyes off the screen) keeps me balanced.”
There are a whole lot of concepts for the Apple Vision Pro that involve displaying video or images in rectangles within a space. It’s basically wearing a headset that just lets you pretend you have lots of different screens in your house. LePore’s idea is more advanced, taking full advantage of the capabilities of the headset. He wants to see a companion to a typical F1 broadcast that displays a model of the actual racing circuit on your coffee table. You can look down and see the cars circulating on the track, giving you an instant intuitive idea of the race state as a whole. He calls it a “spatial broadcast.”
Apple Vision Pro
+ Formula 1
= Killer App (?)a story about:
-design & innovation
-racing royalty
-property theft
and more! pic.twitter.com/6MbLKEDqOB— John LePore (@JohnnyMotion) January 22, 2024
“Today, technology is at a point where sci-fi fantasies can come true… but you need to balance the cinematic with the pragmatic,” says LaPore. “To me, this F1 experience is a perfect blend of things that are genuinely useful (and surprisingly natural), while still feeling magical. The “metaverse” (super sarcastic air-quotes) doesn’t have to be meetings that feel like cyberpunk lazer tag.” LOL.
It’s easy to see why this is compelling. You could watch a driver roaring around the track as they try and blast past a competitor with the undercut pit strategy. You could catch passing moves down the field that the main commentary team was ignoring at the time. At a glance, you could spot DRS trains and looming battles that the broadcaster simply doesn’t have time to explain on the main feed. All this is possible because it’s all happening in front of you in real-time. You can understand it so much more easily when you’re not limited to the 2D viewport of a regular broadcast.
This may seem wild but it’s actually apparently quite feasible according to LaPore.
“Between F1’s live data (already accessible to many), and car/track asset generation is very straightforward (see: any modern videogame from iphone-to-PlayStation),” explains LaPore. “If you were doing this with, say football (see ESPN’s ToyStory/NFL broadcast) you are limited in that it is impossible to capture the appropriate details of the human form in real-time. The beautiful thing is that with racecars you need the 3D position of a car, and other nuances like suspension compression & steering angle. With those things— all achievable— you will get 95% of the detail you would see in person.”
It’s kind of like those awesome battle maps you see in old World War II films, with people pushing around little flocks of aircraft and the like. Except, you’re viewing it through a mixed-reality headset that can overlay digital imagery over your view of the real world. And it’s, you know, cars zipping about a circuit. Rather than desperate fighters at the helm of fierce machines as they try to wrest the freedom of Europe from the grasp of tyrants. I watched a few of those films.
It’s easier to do this with F1 than most motorsports, and most sports in general. LePore notes that cars are far easier to animate and render than humans. This eliminates any need for heavy processing of camera data or animation of multiple limbs. F1 already has highly accurate
telemetry and positional data for all the cars on the track, just ready to be dropped into a tabletop-style simulation. The required infrastructure is all there—the data just needs to be fed into the right display engine to enable this all to work.
Indeed, LePore has been working on this idea for quite some time. As a professional in the visual effects industry, and with plenty of experience designing instrumentation and user interfaces, he was perfectly poised to develop this as an entry for the 2018 F1 Innovation Prize. Ultimately, his idea came second to an idea for a virtual assistant app. LePore puts the loss down to his idea being ahead of its time. Regardless, he still got a trip to the Austin Grand Prix and a second-place trophy handed to him by Lewis Hamilton, so it wasn’t a total waste of a weekend.
It will be interesting to see if the idea has legs in the real world. Fundamentally, the tech is there to achieve it. Whether or not the installed base of mixed-reality headset users and clean coffee tables is there to justify the expense is another thing entirely. There’s also the question of what it would feel like to use in practice, and whether it would really add so much to the regular broadcast. Regardless, as motorsports fans, it’s something we’d love to see happen.
“This could work with any variation of spatial / AR technology— even using your ipad as a lens etc. I emphasize Apple Vision Pro because it is clearly the state-of-the-art hardware today, and I do think the full execution of this will require the ambition to make a best-in-class,” says LaPore. “But that’s also part of the my motivation here: If F1 is the most technologically advanced sport, why shouldn’t it set tech-benchmarks in how we experience sports in general? Why not make the experience of watching the sport as innovative as all the tech going into it. To me, that is the best way to grow F1 further.”
Image credits: John LePore, Apple, Imperial War Museum (public domain)
I want to know more about the instrument cluster please…
Okay, but is projecting it on your coffee table (which introduces a number of notable issues) enough better than just having this alternate view of the race on your tablet, where you can look at it the “normal” way, to justify the expense of AR hardware?
I have very little experience with modern VR equipment, but this seems to be a common sticking point for it really taking off. Many of the best VR experiences are also fun without the expense and awkwardness of VR goggles. There are a few exceptions, but not enough for it to become mainstream.
I like this. The data and driver camera views are all available real-time (in theory) on F1TV.
The problem is that I don’t think I’ve ever had a glitch-free streaming experience from F1TV. I generally download the races when they become available and watch the local file.
Dealing with positional data, main video feed and potentially 20 live driver streams would probably require some back end upgrades. I’d also think this would be priced at a higher tier (still not that bad. F1TV is very reasonably priced in the US for now).
Dang, that looks pretty neat! I could see this being cool at the actual track too, where it’s even harder to follow the whole race. The downside would be wearing those big, dorky goggles out in public.
Can! Could! If!
But will Apple/F1/Etc let you?
This sounds like a really novel use for AR. In our work we have used AR to enable prospective customers for buildings decide on colors and options, just like a car manufacturer’s configurator, and then view the building in 3D, rotate it, etc. Then AR on your phone lets you place the building on your own property, walk around it, take photos and even record video to share. You can also reduce the size of the building, place it on a table and do a walk around. The models are so realistic that you can look in a window on one side and see out a window on the other side.
I was thinking something like this could work with Rally racing, from WRC to Dakar to Rallycross.
Remember when 3D was going to be the “next” big thing? Yea I hope VR ends up in the trash in of technology. It has some great use cases but I feel like it’s being forced upon us just because it is new. It’s also expensive, doesn’t work with certain people.
Smells like a money grab,
You know what is a good “VR” experience? one of those racing games in the arcade where yoor seat moves with the race.
Televisions were probably pretty rudimentary back in the 1930’s-50’s. Yes VR is going to stink for a while but it is pretty complex stuff. I can’t wait to see where this is in 20 years.
How on earth is VR being forced upon us?
3D suffered from a common issue with new tech, being pushed before it was polished and being too expensive at the start. Supposedly the last generation were actually a nice experience, but by then most people had tried a subpar experience and weren’t willing to try again. VR has had a slower burn, most people have had fun brief experiences if they haven’t bought in yet, and there are more applications than what a TV can do. I think VR/AR is still not quite polished and it’s still pretty expensive, and I’m still pretty skeptical about this generation of the tech, but it’s getting somewhere and there’s some interesting potential applications once it’s not tethered to the home and moved out of goofy headsets. All that to say, I don’t think they’re quite equivalent fads.
Honestly, I think this would suit Le Mans the best, since it’s more difficult to keep track of all the positions with multiple classes, as opposed to F1 which essentially runs nose to tail.
Please don’t promote this evil pos of object.
Merging AR tech and tabletop experiences? Sounds like it’s time to duel! Or since this is an auto website, what about dangerous card games played on high-speed motorcycles? Yes, 5D’s is the best Yu-Gi-Oh and it’s not even close.
Oh, I’m not familiar with Yu-Gi-Oh and was certain that motorcycles link was a video of somebody in a tank top and shorts, doing 100, playing a gacha/slots game on their bike’s display computer. I’ve seen texting in similar conditions so it’s not totally out of the question.
Someone tell him to make sure to include AI Grid Girls, offering something F1 no longer has would be a huge plus!
Look at the top photo. I like F1 a ton but…holy crap!
If watching it whilst wearing that head thing makes my hair look like that?
I think maybe I will just stick to the TV.
Plus I don’t ever expect to have a spare 4K to blow on that cool head thing. Or a cool hair cut.
YMMV.
As much as I like the idea of augmented reality, I think I’d rather have a coffee table with little servo driven cars circling it based on telematics from the real cars in the F1 race. They could print out accurate racetracks on vinyl to place on the table and the cars could stay aligned with their track via an Xbox Kinect style camera.
Just spitballing. At minimum that would be rad down at my local pub and no VR widgets required.
Giving me all the nostalgia feels for my REALLY shitty slot-car racetrack circa 1984…
The thing about consumer AR is (at least so far for me) there’s nothing I’ve seen that presents me with a reason/need for the AR headset in the first place. No killer app yet. This feels like it’s somewhere heading in the right direction though. It’s a really smart, well conceived use for the tech but I can’t quite put my finger on what it’s missing that would give it the “WOW that’s so fucking cool take my money!” factor. Obviously if I could figure that out I wouldn’t write about it here and instead go make it and fill a swimming pool with money.
All the same this demo does make me feel like someone is gonna nail it soon and kudos to LePore and thanks for turning me on to this demo.
Commercial AR is a whole different kettle of crawdads and the killer apps are already there, but I don’t really need to simulate a complex organic chemistry lab in my living room.
Agreed. This looks really cool, but it’s not enough for me to actually want to go buy one of these units yet.
VR and AR are pretty different things. AR = you see the real world (and some contextual information overlaid on top. Phone cameras and some pretty standard glasses have AR features today.
VR = everything you see is virtual. You need a headset, and don’t see the real world through it.
I’m well aware of the difference, thx. Buying an Apple AR goggle will be a priority the instant an AR guide for rebuilding my TR6’s engine is published.
That’s the exact thing I was talking about w/ how the killer apps for commercial AR are already here. (Ain’t yet the manual you want, but that scenario and especially for training in industrial mechanic situations are already here and why AR stuff is AMAZING for that).
Thinking about it more, even as spendy as the Apple Vision is, if there’s an AR repair manual for your ride for a lot of us that the purchase price would more than pay for itself in small time.
Piston, thanks for bringing the idea up and the cross-over between where AR is being used to great effect (industrial/commercial training) and how there’s a window for that to cross-over to consumer AR when it comes to us weirdo all-things-auto-lovers.
#TimCook are you listening? The gearheads have spoken.