Sony’s 3D Creator brings 3D scanning to the masses
It takes a lot for me to get excited about a mobile phone, but Sony Mobile managed to do exactly that at a demonstration of their new 3D scanning technology. With the roll-out of Android 8.0 Oreo, starting with the Xperia XZ Premium and the new Xperia XZ1 in New Zealand, Sony has launched its 3D Creator software enabling, users use their phones to easily capture faces and objects in 3D.
I love 3D, Sony loves 3D and you should love 3D. Our world is 3D, but due to technical limitations, up until very recently, we, as consumers, have been limited to sharing our lives in 2D only.
That's going all going to change.
3D movies allow us to peer into a filmmaker's 3D world and virtual reality enables us to totally immerse ourselves in a software developer's 3D environment. The next stage is empowering individuals to create their own 3D content.
Imagine recording your memories, family portrait photos, your wedding cake, your sleeping dog and even your lunch in 3D, and then, if you wish, sharing them with your friends on social media.
Sure, 3D scanning has been around for a while, but it's involved expensive scanning equipment, or taking hundreds of photos and waiting for a complex program or a cloud service to process them into a 3D model. So, when Sony Mobile in Sydney showed me one of their Xperia XZ Premium phones scanning objects, faces and heads, real-time in 3D, I was a bit gobsmacked.
The idea is simple enough, if you take enough photos you can use 3D photogrammetry software like RealityCapture and PhotoScan to create 3D models of your subject. Anyone that's done this will know that in practise it's laborious and fraught with difficulties. You are talking a few hours to get from your photos into a something like SteamVR's Destinations for viewing in virtual reality on the HTC Vive.
Without sounding like a Sony Mobile plant, this new update to the Xperia XZ Premium and XZ1 puts all the hard work when creating 3D scans into a Sony mobile phone. Sony's 3D Creator enables users of Xperia XZ Premium and XZ1, updated to Android 8.0 Oreo, to 3D scan objects in minutes. The software comes with helpful wizards individually designed for face scans, head scans and food (!?!) scans. Once you are a pro, you can start using the freeform scan option to scan anything. I'm advised that, for the moment, the technology is limited to something the size of medium-sized dog.
Two things to be mindful about when scanning. The first is to ensure that you are in a bright, evenly-lit area. Shadowy subjects don't scan well and the Xperia will tell you off if you attempt and to do so. Secondly, when under pressure to carefully scan your subject, be mindful that the Xperia's sleek glass exterior becomes very slippery in clammy hands. I was very fearful of losing my grip on the phone and having it shatter into a million pieces on the stone floor.
The 3D Creator software does it's best to guide you through the scanning process, but it still takes a bit of practise. Basically, all you need to do in order to make a 3D scan is carefully move the phone's camera around your subject, following some handy on-screen guides. You then sculpt in some of the detail that the initial scan may have missed and them click done. In moments, and I really mean moments, you are presented with a 3D scan of your subject which you can rotate and zoom on the screen.
But then what do you do with it?
From the 3D Creator application, you can print your scan using 3D printer. Fancy treating your family to a miniature bust of your head for Christmas? Now you can.
You can impress your friends by creating your own inch-high 3D avatars. 3D Creator comes with its own augmented reality app that enables you to project tiny customised avatars based you your head scans onto your desk, your table or anything else around you. I found it a little fiddly – accidentally pressing buttons on the phone – and, like most AR, it's a bit gimmicky, but OK for a giggle.
The 3D Creator app also allows you to set your model as a live wallpaper on your phone or just share it as a file. The software can output to Wavefront .obj files that can be imported into most industry 3D applications. Alternatively, you can upload your scanned model to popular 3D site, Sketchfab and view it there, or within the Sketchfab VR app.
At the Sony Mobile demonstration at their Sydney HQ, I was shown scans created with 3D Creator exported into free 3rd party app, Shadow Avatar, from Uraniom. This app can take your scanned head and place it on the shoulders of a 3D avatar. You can then choice from a number of options, whereby "you" get to dance around or insert yourself in scenes in the style of Star Wars and even the Blade Runner 2029 movie trailer. You can then export these as mpeg files to keep or share.
This is, of course just the beginning of a new way for people to interact, create and share 3D content with one another.
The obvious application for Sony's 3D scanning technology is taking 3D photos, archiving and preserving memories. It won't be long before people start posting 3D scans of their lunch on social media for friends to pan and zoom around. Thankfully the scanning process is so quick, avoiding food getting to cold whilst capturing it in 3D.
Of course, thinking outside the box, the ability to recreate objects and things, easily in 3D, opens up all sorts of opportunities.
One of the most amazing things that I have ever done in virtual reality was to model a spaceship that I drew as an eight-year old in 3D and inject it into VR. Using my HTC Vive VR set-up I could stand and look up at the huge spaceship and take a walk across the top of its hull.
Scale is nothing in virtual space. Imagine making a scan of the pet dog sleeping, a favourite ornament, a bowl of cornflakes or even Granma's head, and making it ten-metres high so you can stand next to it and look up and admire it in it's full VR glory. If you've got an HTC Vive VR set-up you can do that right now. I'd imagine it won't be long before you can view your 3D Creator models using Sony's own PlayStation VR kit.
More serious applications future could be in aiding medical examinations, criminal investigations and archaeology. Imagine a palaeontologist being able to scan a fossil, in-situ, for study later or a sculptor recording the iterations of their work.
But, what about the Sony Xperia XZ Premium phone itself? I'm not much of a phone guy, but I can't help but be impressed by a phone with a 5.5-inch screen with the same spec as my new TV. The Xperia XZ Premium is the world's first 4K HDR phone using the same Sony Bravia technology as Sony TVs.
Practically speaking, watching the 4K stream of Marvel's Punisher via Netflix with that sort of pixel density is pretty amazing. Of course, the general clarity of the screen for day-to-day use is also fantastic. Opening something like a large spreadsheet, you can actually still see the data without having to zoom right in and pan around.
The Xperia XZ Premium also features a mind-blowing Motion Eye camera capable of recording 960 frames-per-second in 720p. By slowing down the video can capture fast moving object in crisp, blur-free detail. For regular frame-rates you are still able to capture video at 4K! On a phone!
If you are a PlayStation 4 owner to can use your Xperia as a screen to play anywhere in your house via Remote Play. The phone's 4K screen gives you an uncompromising alternative display for your game.
All this is made possible by the ARM-based Qualcomm Snapdragon 835 chip, which won't make your 7th Gen Intel CPU worried, but in terms of mobile devices, it's a powerhouse. In comparison, it offers roughly a third of the performance of an Intel i7 7700K equipped PC with a Geforce GTX1050- which isn't bad considering it can fit the phone in your pocket.
The Sony Xperia XZ1 features the same Snapdragon 835, CPU Adrono 540 GPU, 19MP and 1/2.3" Sony IMX400 camera as the Xperia XZ Premium. The only compromise is that it has a 1080p HDR display rather than the 4K HDR screen.
Sony Mobile has really hit the ball out of the park. Not only is the Xperia XZ Premium an amazing phone, the addition of 3D Creator gives it a killer app.
The ability to scan objects in 3D so easily from your mobile phone is a game changer. We can all look forward to world full of user-created 3D content in the months ahead as this fantastic technology develops.
