10 November 2017

Fused Video Stabilization on the Pixel 2 and Pixel 2 XL




One of the most important aspects of current smartphones is easily capturing and sharing videos. With the Pixel 2 and Pixel 2 XL smartphones, the videos you capture are smoother and clearer than ever before, thanks to our Fused Stabilization technique based on both optical image stabilization (OIS) and electronic image stabilization (EIS). Fused Video Stabilization delivers highly stable footage with minimal artifacts, and the Pixel 2 is currently rated as the leader in DxO's video ranking (also earning the highest overall rating for a smartphone camera). But how does it work?

A key principle in videography is keeping the camera motion smooth and steady. A stable video is free of the distraction, so the viewer can focus on the subject of interest. But, videos taken with smartphones are subject to many conditions that make taking a high-quality video a significant challenge:

Camera Shake
Most people hold their mobile phones in their hands to record videos - you pull the phone from your pocket, record the video, and the video is ready to share right after recording. However, that means your videos shake as much as your hands do -- and they shake a lot! Moreover, if you are walking or running while recording, the camera motion can make videos almost unwatchable:
Motion Blur
If the camera or the subject moves during exposure, the resulting photo or video will appear blurry. Even if we stabilize the motion in between consecutive frames, the motion blur in each individual frame cannot be easily restored in practice, especially on a mobile device. One typical video artifact due to motion blur is sharpness inconsistency: the video may rapidly alternate between blurry and sharp, which is very distracting even after the video is stabilized:
Rolling Shutter
The CMOS image sensor collects one row of pixels, or “scanline”, at a time, and it takes tens of milliseconds to goes from the top scanline to the bottom. Therefore, anything moving during this period can appear distorted. This is called the rolling shutter distortion. Even if you have a steady hand, the rolling shutter distortion will appear when you move quickly:
A simulated rendering of a video with global (left) and rolling (right) shutter.
Focus Breathing
When there are objects of varying distance in a video, the angle of view can change significantly due to objects “jumping” in and out of the foreground. As result, everything shrinks or expands like the video below, which professionals call “breathing”:
A good stabilization system should address all of these issues: the video should look sharp, the motion should be smooth, and the rolling shutter and focus breathing should be corrected.

Many professionals mount the camera on a mechanical stabilizer to entirely isolate hand motion. These devices actively sense and compensate for the camera’s movement to remove all unwanted motions. However, they are usually expensive and cumbersome; you wouldn’t want to carry one every day. There are also handheld gimbal mounts available for mobile phones. However, they are usually larger than the phone itself, and you have to put the phone on it before start recording. You’d need to do it fast before the interesting moment vanishes.

Optical Image Stabilization (OIS) is the most well-known method for suppression of handshake artifacts. Typically, in mobile camera modules with OIS, the lens is suspended in the middle of the module by a number of springs and electromagnets are used to move the lens within its enclosure. The lens module actively senses and compensates for handshake motion at very high speeds. Because OIS responds to motion rapidly, it can greatly suppress the handshake blur. However, the range of correctable motion is fairly limited (usually around 1-2 degrees), which is not enough to correct the unwanted motions between consecutive video frames, or to correct excessive motion blur during walking. Moveover, OIS cannot correct some kinds of motions, such as in-plane rotation. Sometimes it can even introduce a “jello” artifact:
The video is taken by Pixel 2 with only OIS enabled. You can see the frame center is stabilized, but the boundaries have some jello-like artifacts.
Electronic Image Stabilization (EIS) analyzes the camera motion, filters out the unwanted parts, and synthesizes a new video by transforming each frame. The final stabilization quality depends on the algorithm design and implementation optimization of these stages. In general, software-based EIS is more flexible than OIS so it can correct larger and more kinds of motions. However, EIS has some common limitations. First, to prevent undefined regions in the synthesized frame, it needs to reduce the field of view or resolution. Second, compared to OIS or an external stabilizer, EIS requires more computation, which is a limited resource on mobile phones.

Making a Better Video: Fused Video Stabilization
With Fused Video Stabilization, both OIS and EIS are enabled simultaneously during video recording to address all the issues mentioned above. Our solution has three processing stages as shown in the system diagram below. The first processing stage, motion analysis, extracts the gyroscope signal, the OIS motion, and other properties to estimate the camera motion precisely. Then, the motion filtering stage combines machine learning and signal processing to predict a person’s intention in moving the camera. Finally, in the frame synthesis stage, we model and remove the rolling shutter and focus breathing distortion. With Fused Video Stabilization, the videos from Pixel 2 have less motion blur and look more natural. The solution is efficient enough to run in all video modes, such as 60fps or 4K recording.
Motion Analysis
In the motion analysis stage, we use the phone’s high-speed gyroscope to estimate the rotational component of the hand motion (roll, pitch, and yaw). By sensing the motion at 200 Hz, we have dense motion vectors for each scanline, enough to model the rolling shutter distortion. We also measure lens motions that are not sensed by the gyroscope, including both the focus adjustment (z) and the OIS movement (x and y) at high speed. Because we need high temporal precision to model the rolling shutter effect, we carefully optimize the system to ensure perfect timestamp alignment between the CMOS image sensor, the gyroscope, and the lens motion readouts. A misalignment of merely a few milliseconds can introduce noticeable jittering artifact:
Left: The stabilized video of a “running” motion with a 3ms timing error. Note the occasional jittering. Right: The stabilized video with correct timestamps. The bottom right corner shows the original shaky video.
Motion Filtering
The motion filtering stage takes the real camera motion from motion analysis and creates the stabilized virtual camera motion. Note that we push the incoming frames into a queue to defer the processing. This enables us to lookahead at future camera motions, using machine learning to accurately predict the user’s intention. Lookahead filtering is not feasible for OIS or any mechanical stabilizers, which can only react to previous or present motions. We will discuss more about this below.

Frame Synthesis
At the final stage, we derive how the frame is transformed based on the real and virtual camera motions. To handle the rolling shutter distortion, we use multiple transformations for each frame. We split the the input frame into a mesh and warp each part separately:
Left: The input video with mesh overlay. Right: The warped frame, and the red rectangle is the final stabilized output. Note how the non-rigid warping corrects the rolling shutter distortion.
Lookahead Motion Filtering
One key feature in the Fused Video Stabilization is our new lookahead filtering algorithm. It analyzes future motions to recognize the user-intended motion patterns, and creates a smooth virtual camera motion. The lookahead filtering has multiple stages to incrementally improve the virtual camera motion for each frame. In the first step, a Gaussian filtering is applied on the real camera motions of both past and future to obtain a smoothed camera motion:
Left: The input unstabilized video. Right: The smoothed result after Gaussian filtering.
You’ll notice that it’s still not very stable. To further improve the quality, we trained a model to extract intentional motions from the noisy real camera motions. We then apply additional filters given the predicted motion. For example, if we predict the camera is panning horizontally, we would reject more vertical motions. The result is shown below.
Left: The Gaussian filtered result. Right: Our lookahead result. We predict that the user is panning to the right, and suppress more vertical motions.
In practice, the process above does not guarantee there is no undefined “bad” regions, which can appear when the virtual camera is too stabilized and the warped frame falls outside the original field of view. We predict the likelihood of this issue in the next couple frames and adjust the virtual camera motion to get the final result.
Left: Our lookahead result. The undefined area at the bottom-left are shown in cyan. Right: The final result with the bad region removed.
As we mentioned earlier, even with OIS enabled, sometimes the motions are too large and cause motion blur in a single frame. When EIS is further applied to further smooth the camera motion, the motion blur leads to distracting sharpness variations:
Left: Pixel 2 with OIS only. Right: Pixel 2 with the basic Fused Video Stabilization. Note that sharpness variation around the “Exit” label.
This is a very common problem in EIS solutions. To address this issue, we exploit the “masking” property in the human visual system. Motion blur usually blurs the frame along a specific direction, and if the overall frame motion follows that direction, the human eye will not notice it. Instead, our brain treats the blur as a natural part of the motion, and masks it away from our perception.

With the high-frequency gyroscope and OIS signals, we can accurately estimate the motion blur for each frame. We compute where the camera pointed to at both the beginning and end of exposure, and the movement in-between is the motion blur. After that, we apply a machine learning algorithm (trained on a set of videos with and without motion blur) to map the motion blurs in past and future frames to the amount of real camera motion we want to keep, and blend the weighted real camera motion with the virtual one. As you can see below, with the motion blur masking, the distracting sharpness variation is greatly reduced and the camera motion is still stabilized.
Left: Pixel 2 with the basic Fused Video Stabilization. Right: The full Fused Video Stabilization solution with motion blur masking.
Results
We have seen many amazing videos from Pixel 2 with Fused Video Stabilization. Here are some for you to check out:
The videos are taken by two Pixel 2 phones mounted on a single hand grip. Fused Video Stabilization is disabled in the left one.
The videos are taken by two Pixel 2 phones mounting on a single hand grip. Fused Video Stabilization is disabled in the left one. Note that the videographer jumped together with the subject.
Fused Video Stabilization combines the best of OIS and EIS, shows great results in camera motion smoothing and motion blur reduction, and corrects both rolling shutter and focus breathing. With Fused Video Stabilization on Pixel 2 and Pixel 2 XL, you no longer have to carefully place the phone before recording, hold it firmly over the entire recording session, or carry a gimbal mount everywhere. The recorded video will always be stable, sharp, and ready to share.

Acknowledgements
Fused Video Stabilization is a large-scale effort across multiple teams in Google, including the camera algorithm team, sensor algorithm team, camera hardware team, and sensor hardware team.


Mysterious ‘green line of death’ appears on some iPhone X displays


 It wouldn’t be the launch of a first-generation Apple product without a few hiccups — who can forget Antennagate and Bendgate? — and the iPhone X is no different. A handful of users report that a mysterious and inextinguishable green line has appeared on their device’s display. Read More

Read Full Article

Mysterious ‘green line of death’ appears on some iPhone X displays


 It wouldn’t be the launch of a first-generation Apple product without a few hiccups — who can forget Antennagate and Bendgate? — and the iPhone X is no different. A handful of users report that a mysterious and inextinguishable green line has appeared on their device’s display. Read More

Read Full Article

Is a Free PDF Editor Good Enough? Adobe Acrobat Pro DC vs. PDFescape


free-pdf-editor-adobe

Adobe Acrobat Pro DC is the brand’s newest PDF editor. Advertised as an all-encompassing solution, it seems great for heavy-duty PDF users. But it also costs $14.99/month after a free, week-long trial. If you’re an equally cost-conscious and tech-savvy person, you may be wondering about free alternatives. A Free Adobe Acrobat Pro DC Alternative One Adobe alternative that I’ve used in the past is PDFescape. For this post, I’m going to compare PDFescape to Adobe Acrobat Pro DC, just to see if Adobe really is better. PDFescape is available in three versions, including PDFescape Basic. This version is free, and...

Read the full article: Is a Free PDF Editor Good Enough? Adobe Acrobat Pro DC vs. PDFescape


Read Full Article

Looking to Do Some Shopping? Here Are Some Great Deals for You


Today, you can get great deals on a waterproof Bluetooth speaker, a carpet cleaner, a TomTom fitness watch, PC gaming accessories, and much more! TomTom Spark 3 Cardio TomTom Spark 3 Cardio + Music, GPS Fitness Watch + Heart Rate Monitor + 3GB Music + Bluetooth Headphones (Black, Small) TomTom Spark 3 Cardio + Music, GPS Fitness Watch + Heart Rate Monitor + 3GB Music + Bluetooth Headphones (Black, Small) Bluetooth Music Player & Wireless Headphones: Included Bluetooth sport headphones have contoured ear hooks, are sweat and waterproof, last for up to 4 hours, and pair with your Spark and...

Read the full article: Looking to Do Some Shopping? Here Are Some Great Deals for You


Read Full Article

Top 10 Most Popular Accounts on Twitter: Should You Follow Them Too?


most-popular-twitter-accounts

With hundreds of millions of accounts on Twitter, how do you know which ones are worth following? We’ve shared 20 essential accounts that everyone should follow, but there are so many more. Recently, we looked at the top 10 most-followed Instagram accounts. This time, we’re going to do the same for the most popular Twitter accounts. Let’s find out who the biggest names on Twitter are, and whether they’re worth following. 10. Twitter (@Twitter) Followers: 62.1 million | Following: 169 | Tweets: 6,222 The list begins with Twitter’s own account. Of course, this handles official Twitter business, so its tweets...

Read the full article: Top 10 Most Popular Accounts on Twitter: Should You Follow Them Too?


Read Full Article

Apple’s iOS App Store adds weekend deals to boost app downloads


 With the introduction of iOS 11, Apple redesigned the App Store with a heavier focus on editorial content, including articles, interviews, how-to’s, lists and other round-ups with app recommendations. Now, it’s adding another feature, as well, called “This Weekend Only.” The new section lists deals that can be found inside apps, as opposed to discounts on downloads of… Read More

Read Full Article

Amazon expands its Influencer Program to include Twitter and Instagram, in addition to YouTube


Amazon is expanding its Influencer Program beyond YouTube to also include Twitter and Instagram, the company announced on Thursday. First launched into beta earlier this spring, the program initially targeted YouTube stars by offering them a way to make money from the products they promoted in their videos through an affiliate-like relationship with Amazon. Many of today’s influencers… Read More
Read Full Article

How Every Operating System Keeps You Safe


os-keeps-safe

Operating systems are responsible for being the go-between for allowing programs to run on your computer. They divide up your computer resources, splitting them up between applications, to let you manipulate the hardware underneath. Put shortly, any machine that can run programs needs one, whether that be macOS, Windows, or Linux. While some operating systems are more secure than others, many modern ones these days do their best to protect themselves from potential problems out of the box. As such, it would be erroneous to say that just because some are known to have more exploits than others that they...

Read the full article: How Every Operating System Keeps You Safe


Read Full Article

Facebook relaunches Events app as Facebook Local, adds bars and food


 Bad news for Yelp and good news for nightlife lovers. When you want to go out, it doesn’t necessarily have to be to an event. So to help you discover bars, restaurants, and nearby attractions too, Facebook is rebranding its standalone Events app as “Facebook Local”. Launching in the U.S. today on iOS and Android, Facebook Local combines events and permanent places to a… Read More
Read Full Article

Facebook relaunches Events app as Facebook Local, adds bars and food


 Bad news for Yelp and good news for nightlife lovers. When you want to go out, it doesn’t necessarily have to be to an event. So to help you discover bars, restaurants, and nearby attractions too, Facebook is rebranding its standalone Events app as “Facebook Local”. Launching in the U.S. today on iOS and Android, Facebook Local combines events and permanent places to a… Read More

Read Full Article

Pop!_OS Has Arrived: How Does It Compare to Ubuntu?


popos-vs-ubuntu

System76 is one of the most well-known hardware companies in the free and open source software world. That’s not to say the brand is by any means a household name. Nonetheless, System76 has been selling computers that run Ubuntu for over a decade. That’s why the company made news when it announced that it would stop shipping Ubuntu and would instead provide its own Linux-based operating system, Pop!_OS. In the past few weeks, the first official release of Pop!_OS became available for download. Now it’s shipping as an option on new computers from System76. Should you check it out? How...

Read the full article: Pop!_OS Has Arrived: How Does It Compare to Ubuntu?


Read Full Article

How to Hide Files, Folders, and Drives in Windows 10


hide-files-folders-drives-windows

If you share your computer with others, you may want to hide some files and folders so people don’t go nosing into your files. We’ve talked about password protecting files and folders on Windows. But if that’s overkill for your purposes, you can simply hide your files and folders. We’ve previously covered how to hide files and folders on a Mac. Here are several ways to hide files and folders on Windows. 1. Hide Individual Files/Folders Using the Command Line A simple way to hide a file or folder is to change its attributes using the attrib command on the...

Read the full article: How to Hide Files, Folders, and Drives in Windows 10


Read Full Article

Parrot’s Henri Seydoux comes to Disrupt to talk about commercial drones


 Henri Seydoux first founded Parrot in 1994. But the company has drastically evolved over time and managed to remain relevant more than 20 years later. Ten years ago, Parrot started working on consumer drones. And now, the company is in the middle of another shift as it now focuses more than ever on commercial drone use cases. That’s why we’re excited to announce that Parrot founder… Read More

Read Full Article

Light Up Your Life With Fun LED Projects for Kids and Adults


led-projects

Craft projects have come along way since simple yarn, glue, and papier-mâché. But this is all for the good because crafts with LED lights can be decorative, fun, and easier than you think. For both adults and children, these creative ideas with those lights that last will bring a sparkle to your craft table. Colorful Crafts for Adults Many adults enjoy making their own unique decorations and are serious about their craft hobby. For some bright new ideas with LED lights, take a look at these cool creations. 1. Super Easy Centerpieces Whether it’s for your holiday décor, dinner party,...

Read the full article: Light Up Your Life With Fun LED Projects for Kids and Adults


Read Full Article

Use Amazon Wish and Idea Lists to Get Gifts You Actually Want


Getting gifts you aren’t interested in can be awkward. You want to be grateful to the person who gave you the present, but it’s a pain to return and/or exchange it for something you’ll actually use. If you’d rather guarantee getting something you’ll actually want the first time around, you should take advantage of two of Amazon’s best hidden features: Wish Lists and Idea Lists. Idea Lists allow you to “create collections of interesting items found on Amazon” that you can use yourself or share with others. Wish Lists, on the other hand, are there to help others know what you...

Read the full article: Use Amazon Wish and Idea Lists to Get Gifts You Actually Want


Read Full Article

Find New Travel Destinations With This Nifty Instagram App


Instagrammers and their beautiful travel photos can often be deceptive — and you may not realize until after you’ve bought your plane tickets and made your travel plans. That doesn’t mean you should stop using the visual pull of Instagram photos for your travel inspiration. There are many genuine travel Instagrammers around to stoke your curiosity, those who don’t use Photoshop’s trickery. Instagram has given us a new way to discover new places to travel to. There is also a new way to make this location search and travel planning better. Let’s bring out the little travel guide for your pocket. Use the Sherpa Guide for...

Read the full article: Find New Travel Destinations With This Nifty Instagram App


Read Full Article

How to Delete All Your Tweets


We’ve all heard the horror stories. The woman who lost her job because of a Facebook post she made during a party at college a decade ago. The man who was sacked because he accidentally tweeted something that could be easily misinterpreted. The reality is that our social media personas are now an extension of ourselves. Would-be employers trawl through your data to see if there’s anything concerning. Border guards look at your online identity as part of their visa checks. The list goes on. If you regret a few things you said on Twitter in years gone by, it’s...

Read the full article: How to Delete All Your Tweets


Read Full Article

Can’t Raise iPhone Ringer Volume? Here’s Why


iOS 11 brought a ton of great features and a ton of frustrations, particularly for those using older phone models. One of the slightly frustrating changes, that luckily has an easy fix, is a change to the the phone’s ringer settings. Prior to the update, you could turn the ringer volume up and down simply by using the volume rocker. After the update, the volume rocker only adjusts the phone’s speaker volume. If that has happened to you and you want to restore the older functionality: Go to Settings > Sounds and toggle Change with Buttons on. That’s it – you’ll once again...

Read the full article: Can’t Raise iPhone Ringer Volume? Here’s Why


Read Full Article

Youtube Kids implements new policy to flag inappropriate videos targeted at children


 A New York Times piece and a subsequent Medium post this week highlighted an ongoing problem with Youtube Kids — bizarre and disturbing videos aimed at young children using key words and popular children’s characters. Now Youtube says it is putting in place a new process to age-restrict these types of videos in the main Youtube app. “Age-restricted content is automatically… Read More
Read Full Article

Facebook defends revenge porn pilot that has people upload nude images of themselves


 Facebook Global Head of Safety Antigone Davis has clarified some things about how Facebook’s test pilot to combat revenge porn in Australia works. The strategy entails uploading your nude photos or videos to Messenger in order to help Facebook tag it as non-consensual explicit media. Read More
Read Full Article

Twitch is finally releasing the nearly 30 million usernames associated with Justin.tv


 Justin.tv, the life-streaming service that eventually grew into Twitch, was shut down back in 2014. Since then, many Twitch users have asked when the company would release the old Justin.tv usernames – requests that escalated after Twitch introduced the ability to change usernames earlier this year. As it turns out, Twitch is now doing just that. In an email sent confidentially to… Read More
Read Full Article

UberEATS Now Lets You Rate Restaurants


UberEATS users now have the option to rate everything about their experience, and not just the poor delivery driver doing their bidding. In a world where most things come with either a rating out of 5, or a simple thumbs-up/thumbs-down (including Netflix) this is long overdue. For the uninitiated, UberEATS is Uber’s food delivery service. You open the UberEATS app, pick a local restaurant, and choose some food. And then some poor sap in a car, on a bike, or even on a skateboard (see below) will whisk your food to your door. And now you can rate your experience....

Read the full article: UberEATS Now Lets You Rate Restaurants


Read Full Article

Google’s improved image blending means a more seamless Street View


 Google is hard at work behind the scenes improving one of its ambitious technical projects ever – Street View. The company previously revealed that its been rolling out improved camera cars with better photographic equipment to improve the quality and resolution of images that make up its street-level views in Google Maps, but it’s also fixing the sometimes messy stitching that… Read More
Read Full Article

10 of the Best Large Camera Bags for Fashion, Function, and Price


camera-bags

Outside of the camera itself, the most important investment: camera bags. The right bag makes it easy to find all of your lens and accessories while keeping everything protected. It’ll also make carrying your heavy gear around as pleasant as possible. There are so many different bags out there to choose from, and they vary in price from around $15 all the way up to hundreds of dollars. How are you supposed to choose between so many models with such wide variances in price? We’re here to help! Professional Grade: Lowepro, Pelican, and More If you have a high-end camera,...

Read the full article: 10 of the Best Large Camera Bags for Fashion, Function, and Price


Read Full Article

You Can Now Try Google’s New Files Go App


Google has released its new Files Go app in beta to anyone who wants to give it a spin. Files Go is a file management app designed to clean up space on your phone and help you stay organized. It also lets you share files with other Android users in a way not dissimilar to Apple’s AirDrop. Earlier this week a new Android app called Files Go appeared on Google Play. However, Google pulled the app when tech sites caught wind of it. Now, Google has made Files Go available once more, and even asked for feedback. So it would...

Read the full article: You Can Now Try Google’s New Files Go App


Read Full Article

Essential Phone can now stream live to Facebook from the 360 camera add-on


 The Essential Phone just got a camera software update that will allow owners of the device to broadcast live, 360-degree video right from the native camera app, provided they have the Essential 360 Camera, too. The update requires no extra software, and live streaming becomes available soon as you snap the 360 camera onto the magnetic accessory port on the back of the Essential Phone. Snapping… Read More

Read Full Article