24 October 2020

This Week in Apps: Quibi dies, Snapchat soars, Halide upgrades for iPhone 12


Welcome back to This Week in Apps, the TechCrunch series that recaps the latest OS news, the applications they support and the money that flows through it all.

The app industry is as hot as ever, with a record 204 billion downloads and $120 billion in consumer spending in 2019. People are now spending three hours and 40 minutes per day using apps, rivaling TV. Apps aren’t just a way to pass idle hours — they’re a big business. In 2019, mobile-first companies had a combined $544 billion valuation, 6.5x higher than those without a mobile focus.

Top Stories

Quibi dies…and no one was surprised

There was so much wrong with Quibi’s premise that it’s sometimes hard to even know where to start. But at the core, its problem was that it fundamentally misunderstood how, when and why users would watch video on their phones.

The company’s thinking was that you could fund high-production value content ($100K/minute, yikes) then chop it up into smaller “bites,” add a technology layer, then call this a reinvention of cinema.

The reality is there was little demand for this sort of content, and it didn’t fit with how people want to be entertained on their phones.

When people want to appreciate high-quality filmmaking (or even TV production), they tend to want a bigger screen — they’ve spent money for their fancy high-def or 4K TV, after all. Pre-COVID, they might even pay to go a movie theater. On mobile, the production value of content is far less of a concern, if it even registers.

Quibi also misunderstood what users want to watch in terms of video on their phones when they have a few minutes to kill.

By positioning its app in this space, it had to compete with numerous and powerful sources for “short-form” content — existing apps like YouTube, TikTok, Facebook (e.g. News Feed content, Watch feeds), Instagram Stories, Snapchat and so on. This is content you don’t have to get invested in, since you’re just distracting yourself from a few minutes of boredom. It’s not a time or place to engage with a longer story — chopped or otherwise.

Quibi also cut the length of content to serve its artificial limitations — at the expense of story quality and enjoyment.

A reality show dumbed down to just its highlights is almost unwatchable, as it exposes the editors’ machinations and manipulations that are better hidden among longer stretches of fluff. And there was simply no reason to cut down movies — like Quibi’s “The Dangerous Game,” for example — into pieces. It didn’t elevate the storytelling; it distracted from it. And if you wanted a quick news update (e.g. Quibi’s “Daily Essentials”), you didn’t need a whole new app for that.

Quibi content may have been considered “high quality,” but it often wasn’t good. (I still can’t believe I sat through an episode of “Dishmantled,” where chefs had to recreate dishes of food that were thrown in their face. And Quibi had the nerve to shame YouTube’s low-quality and lack of talent?!)

Quibi also wanted to charge for its service, but its catalog wasn’t designed for families, with content that ranged from kids to adult programming. It didn’t offer parental controls. This immediately limited its competitiveness.

At launch, Quibi also limited itself to the phone, which meant it limited your ability to use the phone as a second screen while you watched a show. (There was no PiP support). TechCrunch has been writing about phones as the second screen for the better part of a decade, often with a focus on startups. But in Quibi’s case, it killed the second screen experience, seemingly forgetting that people text friends, order food, check Twitter and peek in on other apps while a TV show plays in the background. Did it really think that a reboot of “Punk’d” deserved our full attention?

Quibi naturally blamed COVID for its failure to thrive. It had imagined a world where users had ample time to kill while out and about: commuting on the subway, standing in long lines, that sort of thing.

But even this premise was flawed. It would have eventually caught up to Quibi, too; COVID just accelerated it. The issue is that Quibi imagined the U.S. as only a swath of urban metros where public transportation is abundant and standing in lines is the norm. In reality, more than half (52%) the U.S. is described as suburban, 27% is urban and 21% is rural. Non-urban commuters often drive themselves to work. Sure, they could stream Quibi during those commutes, but not really look at it. So why burn high-production value on them? And standing in long lines, believe it or not, is not actually that common in smaller cities and towns, either. If it only takes two minutes to grab a coffee or a burrito before you hop back in your car, do you really want to start a new show?

So where would that have left Quibi? Hoping for Gen Z’ers attention as they lounge around their bedrooms looking for something to do? And yet it wanted to appeal to these kids using Hollywood A-Listers they don’t even know? As COVID pressed down, it left Quibi in competition with (often arguably better) content that streamed natively on the TV from apps like Netflix, HBO, Hulu, Prime Video, Disney+, and others where you could binge through seasons at once instead of waiting every week for a new “quick bite” to drop.

There’s more, so much more that could still be said, including the fact that a former eBay and HP CEO may not be the right person to lead a company that wanted to dazzle a younger demographic. Or how its video-flipping TurnStyle feature was clever, but added complexity to filmmaking, and was not enough of a technological leap to build a business around. Or how, no matter how much money it had raised, it was still not enough, compared with the massive budgets of competitors like Netflix and Amazon.

You can read a further post-mortem round-up here. And another here. Because we can’t get enough post-mortems, apparently.

In the meantime, TikTok still isn’t banned.

Snap hits record $50B valuation

Snapchat’s maker was forecast to bring around $555 million in revenues in Q3 but posted $679 million instead, a 52% YoY increase, in a surprise earnings beat. EPS were an adjusted $0.01, beating an expected loss of $0.04. The company also grew daily active users by 4% (11 million) to 249 million, an 18% YoY increase. Snap’s net loss of $200 million was a 12% improvement over last year, too.

As a result of the earnings, shares jumped nearly 30% the next day and its valuation cracked $50 billion for the first time, a record high.

During earnings, the company touted it now reaches 90% of the Gen Z population and 75% of millennials in the U.S., U.K. and France. User growth was attributed to new products, including Profiles, Minis, Lens creation tools and AR ads. In particular, Snap leveraged the Facebook ad boycott to reach out to brands that wanted to “realign their marketing efforts” with companies that “share their corporate values,” the company said.

Snap also just launched its TikTok competitor, Sounds on Snapchat, which lets users add licensed music to their Stories.

Weekly News Round-Up

Platforms

  • Apple releases iOS and iPadOS 14.1. The first major update to iOS 14 delivers multiple bug fixes, including those impacting widgets, streaming video and Family Setup on Apple Watch, among others. It also added support for 10-bit HDR video playback and editing in Photos on iPhone 8 and later.
  • iOS 14 bug continues to reset default email and browser apps. After updating your preferred email or browser app, iOS 14 forgets what third-party app you’ve set as the default. Yes, it was doing this before. Are we still so sure it’s a bug?
  • DOJ antitrust lawsuit goes after the multibillion-dollar deal that positioned Google as the default search engine on browsers, phones and other Apple devices.
  • AirTags patent applications describe use cases like locating the nearest defibrillator, monitoring users’ posture and playing avatar-based games, giving a little more insight into how Apple envisions the future of its smartphone-findable tags.
  • Google embraces iOS 14 widgets. Google already offered one of the more useful widgets for iOS 14 with its Search widget, which has been downloaded by “millions.” This week, it introduced more, including a Google Photos widget that let you revisit your memories, and a YouTube Music widget.
  • RCS support in Android Messages expands. Following the U.S. debut, RCS has rolled out to a number of new countries, and can now be found in Italy, Portugal, Singapore, Argentina, Pakistan, Poland, Turkey, Denmark, Netherlands, Austria, Bangladesh, Belgium, Croatia, Czechia, Greece, Ireland, Israel, Kosovo, Lithuania, New Zealand, Serbia, Slovenia, Sri Lanka, Switzerland, Australia, Bulgaria, Indonesia, Japan, Kenya, Latvia, Lebanon, Uganda and Ukraine. The last nine were just this month.

Trends

Image Credits: Sensor Tower

  • Buy Now, Pay Later app usage in the U.S. up 186% year-over-year as of Sept. According to Sensor Tower, apps that let consumers make purchases on payment plans have been climbing steadily this year since the COVID-19 pandemic. The report looked at Klarna, Affirm, Afterpay and QuadPay, which together have generated 18 million lifetime installs across the App Store and Google Play. Installs were up 115% YoY in September, while monthly actives were up 186%.
  • U.S. contact-tracing apps are a disjointed wreck. The WSJ examined the state of COVID-19 contact-tracing apps in the U.S. and found that states focusing on their own efforts, due to the lack of a national plan, has left a disjointed patchwork of tools. Only 10 states, plus D.C., have used the framework built by Google and Apple; 11 are piloting or building apps. The EU, meanwhile, switched on cross-border interoperability for its first batch of tracing apps.
  • Gen Z spends 10% more time using top non-game apps than older users, at 4.1+ hours per month. The figure excludes pre-installed apps and was calculated on Android devices in select markets, including the U.S. Gen Z users also engaged with non-game apps more often than older users, at 120 sessions per month per app.
  • U.S. consumers spend $20.78/mo on average on their app subscriptions, according to new data from Adjust. The 25 to 34-year-old age group spends the most on subscription apps at $25.85/mo, while those 55 and over spend the least, at $13.97/mo. In addition, more than a quarter of millennials and Gen Z consumers said they have stopped paying for other services in order to buy subscriptions on mobile app services (e.g. option for fitness apps over going to the gym).
  • Dating apps are on the rise in the U.S., says Apptopia. New users for Hily, Match, BLK, Bumble and Grindr are on pace to grow month-over-month at 32%, 28%, 20%, 18% and 11%, respectively.

 

Services

  • Amazon’s Luna game streaming service opens in early access to its first customers. The service offers a library of 50 games and works on Mac, PC, Amazon Fire TV, and iOS devices, courtesy of a web app to work around the App Store rules. Initial reviews describe the service as sometimes struggling with performance over Wi-Fi, but offering a good web app experience. Luna features some big titles but xCloud still has the better lineup. Its real killer feature, however, may be the promised Twitch integration, arriving in the future.
  • SoundCloud launches a $19.99/month DJ plan, SoundCloud DJ, that offers unlimited offline access to its catalog. Users can also stream high-quality audio and mix tracks using select DJ apps, including Virtual DJ, Cross DJ and Denon DJ.
  • Put your five-star reviews on your home screen. IMore spotted a must-have motivational tool for developers: a way to put your app’s five-star reviews as a widget on your home screen; $1.99 for this happiness boost.

Security/Privacy

Deadpool

  • Apple quietly discontinues its Apple TV Remote app. The app was removed from the App Store on Wednesday. Users are now expected to use the Remote feature built into the Control Center since iOS 12 instead.
  • Google will end support for its location-sharing Trusted Contacts app in December, and removes it from the Play Store. Users are directed to use similar features in Google Maps instead for finding friends and family.

Policies and Politics

  • Coalition for App Fairness more than doubles a month after its debut. The Coalition for App Fairness (CAF), a newly formed advocacy group pushing for increased regulation over app stores, has more than doubled in size with this week’s announcement of 20 new partners. The organization, led by top app publishers and critics, including Epic Games, Deezer, Basecamp, Tile, Spotify and others, debuted in late September to fight back against Apple and Google’s control over app stores, and particularly the stores’ rules around in-app purchases and commissions.

App News

  • Facebook to increase investments in WhatsApp for business. The company said it will expand Shopping on WhatsApp and will charge businesses for some of the services it offers on the chat app, in order to grow revenues. This includes offering to manage businesses’ WhatsApp messages via Facebook’s own hosting services. Facebook offered this info as more of a look into its roadmap, but without specifics on new services or pricing.
  • Facebook is cloning Nextdoor. The feature is in testing in Canada and sees Facebook automatically generating neighborhood groups to connect local users with people, activities and items for sale.
  • Court approves Kik’s settlement with SEC. The ruling ends a multi-year court battle by allowing Kik to pay a one-time $5 million fine for its violation of securities law for failing to register its 2017 distribution of its Kin tokens in its ICO.
  • Roblox passes $2B in mobile player spending ahead of its planned IPO. The company’s revenues, accelerated by the pandemic, crossed the $1.5 billion mark in May 2020, then picked up another $500 million in five months, says Sensor Tower.
  • Cameo enters B2B sales. The custom celebrity video app repositions its business of personalized greetings for B2B sales through an integration and rev share agreement with corporate gifting platform Sendoso.
  • Adobe adds a chain of custody tool in the beta release of Photoshop and Behance that will fight misinformation and keep content attributed properly.
  • Stitcher’s podcasts come to Pandora as acquisition completes. The Stitcher app also got a revamp following the deal’s finalization. The move brought several bigger podcast titles in house, thanks to Earwolf, including “Freakonomics Radio,” “My Favorite Murder,” “SuperSoul Conversations from the Oprah Winfrey Network,” “Office Ladies,” “Conan O’Brien Needs a Friend,” “Literally! with Rob Lowe,” “LeVar Burton Reads” and “WTF with Marc Maron.”
  • NYT has an iOS 14 widget now. The new widget will put NYT headlines on your home screen. Note that while the widget can be installed by anyone, if you want to click through to read, you’ll still need to be a subscriber.
  • PicsArt brings its app-based design tools to the web. The creative platform is chasing business users with the launch of its AI tools on picsart.com. The debut suite includes a template editor, background and object remover, video slideshow maker, text editor, and others.

Funding and M&A

  • Chinese tutoring app Yuanfudao has raised $2.2 billion from investors, surpassing Byju’s as the most valuable edtech company in the world, as it’s now worth $15.5 billion.
  • Retool raises $50M in funding, led by Sequoia, for its low-code tools for building internal apps that work on either desktop or mobile. The new round values the business at nearly $1 billion. Other backers include GitHub CEO Nat Friedman, Stripe founders Patrick and John Collison, Brex Inc. founders Henrique Dubugras and Pedro Franceschi and Y Combinator co-founder Paul Graham.
  • Syte raises $40M to bring visual shoppers to e-commerce retailers. Visual search is already popular in apps like Google, Pinterest and eBay, but Syte wants under retailers to have the option. The round was led by return investor Viola Ventures.
  • 98point6 raises $118M for its AI-powered telemedicine platform that works on web and mobile (iOS and Android).

Recommended Downloads

Halide Mark II

Image Credits: Lux

The developers of popular pro iPhone camera apps Halide and Spectre this week launched their latest creation, the Halide Mark II camera app. The new interface has been designed for one-handed operation and includes a range of new features.

These include a new gesture-based automatic and manual switcher; tactile touch for enabling and disabling features like exposure warnings, focus peaking, and loupe as you adjust exposure or focus; an overhauled manual mode; new dynamic labeling of controls and actions to explain features to new users; support for the edge-to-edge interface of the iPhone 12 models; a redesigned reviewer with a full metadata read-out; in-app memberships for photo lessons; and over 40 more changes.

A new “Coverage” feature can take a photo with Smart HDR 2/3 and Deep Fusion for maximum quality and computational processing as well as a RAW file — with only a slight delay between captures.

Image Credits: Lux

Halide Mark II also uses machine learning to process an iPhone RAW file in the app (ProRAW) with 17 steps, including detail enhancement, contrast and color adjustment and more. This feature, called Instant RAW, intelligently develops the file to get the best possible results.

And the app includes top pro tools, like a new waveform and color exposure warnings (zebras) that use XDR (Extended Dynamic Range) 14-bit RAW sampling, for accurate exposure previews and readings.

The app is $36 (currently $30 during a promo period) if you want to only pay once. Otherwise it’s $11.99 per year on subscription (currently $9.99 per year if you lock in the price now during the promo period). Subscribers to the membership plan also get perks, like custom icons. Existing Halide 1 users, unbelievably, are upgraded for free but are asked to support the app with a membership.

ClipDrop — AR Copy Paste

A new app called ClipDrop launches on iOS, Android, macOS and Windows as a new sort of “copy and paste” experience. The app uses state-of-the-art vision AI to copy images from your desktop with a screenshot to any other app (e.g. Docs, Photoshop, Canva, etc.) and it allows you to extract anything — objects, people, drawings or text.

The mobile app lets you snap photos of real-world items and then digitally transfer them to other apps or websites. In the below demo, the company shows how you could “clip” an image of an article of clothing using the camera, then import the photo into a document.

The company also just released a plugin for Photoshop that lets you drop the image into its app as a new layer with an editable mask.

The app is $39.99 per year (until November 2020, when it ups to $79.99 per year.)

Adobe Illustrator on iPad + Adobe Fresco on iPhone

Image Credits: Adobe

As part of Adobe’s virtual MAX 2020 conference this week, the company launched the first public version of its Illustrator vector graphics app on the iPad and brought its Fresco drawing and painting app to the iPhone. In time, the company plans to bring more effects, brushes and AI features to Illustrator. Fresco 2.0, meanwhile, includes new smudge brushes and support for personalized brushes, among other things.

Party Squasher

Designed for landlords, Airbnb owners or other vacation rental property owners, Party Squasher offers a hardware device and paired mobile app that counts the number of people at your house by counting the mobile phones in or around a house. The phones can be counted even if they’re not connected to the home’s Wi-Fi.

Because the device doesn’t include cameras or microphones, it’s ideal for ensuring that renters aren’t hosting large (and these days, potentially illegal) parties without violating privacy.

In the event that a large gathering is present, you’re sent a text or email so you can take action.

The device is $249 and the app charges a $199 per year subscription.

Tweets

 

The No. 1 game in the App Store is now Among Us!.

Can you guess why?


Read Full Article

The Best Online Casino Games Based On Your Personality


When we talk about online casino gameplay, we all have our very own preferences. Some of us enjoy spinning the reels, while others are inclined to table games. The type of game you choose often depicts your personality. As the day ends after a long, tiring work schedule, there’s nothing better than relaxing in a […]

The post The Best Online Casino Games Based On Your Personality appeared first on ALL TECH BUZZ.


What To Know About A Career In Cyber Security?


Are you considering a career in cyber security? This is an industry on the rise right now with cybercrime considered an enormous threat to businesses and individuals. With dangerous threats constantly in development, cyber security is developing at a similar rate. Ensuring that the war on cybercrime continues and will continue to do so for […]

The post What To Know About A Career In Cyber Security? appeared first on ALL TECH BUZZ.


Live News Streaming is Growing Fast – Why You Should You Catch up With It


Live streaming is when a video s broadcasted live, instead of being recorded earlier. Additionally, it is an umbrella term, since it incorporates TV broadcast, accessible in real Internet time. When it comes to live news streaming, it directs at the content that is produced by channels to broadcast online. News channels exhibit live streaming […]

The post Live News Streaming is Growing Fast – Why You Should You Catch up With It appeared first on ALL TECH BUZZ.


Curbside


Curbside

What it takes to create social change against all odds | Ralph Nader

What it takes to create social change against all odds | Ralph Nader

Over his decades-long career as a political activist, Ralph Nader has helped expose some of the greatest misdeeds of large corporations. You may be familiar with the real-world changes his work sparked: the Clean Air Act, automobile safety laws, regulation of the tobacco industry and more. Tracing the arc of his time advocating for change, Nader shares how he helped catalyze social progress against overwhelming odds -- and shows how you can participate in advancing the common good for generations to come.

https://ift.tt/2TmTgP0

Click this link to view the TED Talk

Disrupt! Five ways the internet has broken local barriers


The development of the internet from a thing you would turn on the computer to access to something that is consistently surrounding our every waking moment is one of the most consequential innovations that have happened to the entire world. Before its advent, accessing information was more difficult. Now, with any sort of data one […]

The post Disrupt! Five ways the internet has broken local barriers appeared first on ALL TECH BUZZ.


OnePlus’s 8T handset brings faster charging and a 120Hz display for $749 


OnePlus continues its twice-yearly smartphone cycle with today’s arrival of the 8T. The latest device isn’t a huge upgrade over April’s OnePlus 8, but continues the company’s longstanding tradition of offering some of the most solid Android handsets at a reasonable price point. The cost has edged up a bit in recent years, but $749 is still pretty good for what the 8T offers.

The big updates this time out are the 120Hz refresh rate for its 6.55-inch display and super-fast charging via the Warp Charge 65. That should get the 4,450 mAh of battery capacity up to a full day’s charge in 15 minutes, with a full charge taking a little less than 40 minutes.

There are an abundance of cameras here — four in total. That includes a 48-megapixel main (with built in optical image stabilization), 16-megapixel ultra-wide angle and, more surprisingly, a macro and monochrome lens. The handset joins the even more affordable Nord, which is set to arrive in the U.S. in the near future at a sub-$500 price point.

OnePlus has been undergoing some corporate changes in recent weeks, as well. Co-founder Carl Pei recently announced he will be leaving the company. “These past years, OnePlus has been my singular focus, and everything else has had to take a backseat,” he told TechCrunch. “I’m looking forward to taking some time off to decompress and catch up with my family and friends,” he wrote. “And then follow my heart on to what’s next.”


Read Full Article

23 October 2020

Rethinking Attention with Performers


Transformer models have achieved state-of-the-art results across a diverse range of domains, including natural language, conversation, images, and even music. The core block of every Transformer architecture is the attention module, which computes similarity scores for all pairs of positions in an input sequence. This however, scales poorly with the length of the input sequence, requiring quadratic computation time to produce all similarity scores, as well as quadratic memory size to construct a matrix to store these scores.

For applications where long-range attention is needed, several fast and more space-efficient proxies have been proposed such as memory caching techniques, but a far more common way is to rely on sparse attention. Sparse attention reduces computation time and the memory requirements of the attention mechanism by computing a limited selection of similarity scores from a sequence rather than all possible pairs, resulting in a sparse matrix rather than a full matrix. These sparse entries may be manually proposed, found via optimization methods, learned, or even randomized, as demonstrated by such methods as Sparse Transformers, Longformers, Routing Transformers, Reformers, and Big Bird. Since sparse matrices can also be represented by graphs and edges, sparsification methods are also motivated by the graph neural network literature, with specific relationships to attention outlined in Graph Attention Networks. Such sparsity-based architectures usually require additional layers to implicitly produce a full attention mechanism.

Standard sparsification techniques. Left: Example of a sparsity pattern, where tokens attend only to other nearby tokens. Right: In Graph Attention Networks, tokens attend only to their neighbors in the graph, which should have higher relevance than other nodes. See Efficient Transformers: A Survey for a comprehensive categorization of various methods.

Unfortunately, sparse attention methods can still suffer from a number of limitations. (1) They require efficient sparse-matrix multiplication operations, which are not available on all accelerators; (2) they usually do not provide rigorous theoretical guarantees for their representation power; (3) they are optimized primarily for Transformer models and generative pre-training; and (4) they usually stack more attention layers to compensate for sparse representations, making them difficult to use with other pre-trained models, thus requiring retraining and significant energy consumption. In addition to these shortcomings, sparse attention mechanisms are often still not sufficient to address the full range of problems to which regular attention methods are applied, such as Pointer Networks. There are also some operations that cannot be sparsified, such as the commonly used softmax operation, which normalizes similarity scores in the attention mechanism and is used heavily in industry-scale recommender systems.

To resolve these issues, we introduce the Performer, a Transformer architecture with attention mechanisms that scale linearly, thus enabling faster training while allowing the model to process longer lengths, as required for certain image datasets such as ImageNet64 and text datasets such as PG-19. The Performer uses an efficient (linear) generalized attention framework, which allows a broad class of attention mechanisms based on different similarity measures (kernels). The framework is implemented by our novel Fast Attention Via Positive Orthogonal Random Features (FAVOR+) algorithm, which provides scalable low-variance and unbiased estimation of attention mechanisms that can be expressed by random feature map decompositions (in particular, regular softmax-attention). We obtain strong accuracy guarantees for this method while preserving linear space and time complexity, which can also be applied to standalone softmax operations.

Generalized Attention
In the original attention mechanism, the query and key inputs, corresponding respectively to rows and columns of a matrix, are multiplied together and passed through a softmax operation to form an attention matrix, which stores the similarity scores. Note that in this method, one cannot decompose the query-key product back into its original query and key components after passing it into the nonlinear softmax operation. However, it is possible to decompose the attention matrix back to a product of random nonlinear functions of the original queries and keys, otherwise known as random features, which allows one to encode the similarity information in a more efficient manner.

LHS: The standard attention matrix, which contains all similarity scores for every pair of entries, formed by a softmax operation on the query and keys, denoted by q and k. RHS: The standard attention matrix can be approximated via lower-rank randomized matrices Q′ and K′ with rows encoding potentially randomized nonlinear functions of the original queries/keys. For the regular softmax-attention, the transformation is very compact and involves an exponential function as well as random Gaussian projections.

Regular softmax-attention can be seen as a special case with these nonlinear functions defined by exponential functions and Gaussian projections. Note that we can also reason inversely, by implementing more general nonlinear functions first, implicitly defining other types of similarity measures, or kernels, on the query-key product. We frame this as generalized attention, based on earlier work in kernel methods. Although for most kernels, closed-form formulae do not exist, our mechanism can still be applied since it does not rely on them.

To the best of our knowledge, we are the first to show that any attention matrix can be effectively approximated in downstream Transformer-applications using random features. The novel mechanism enabling this is the use of positive random features, i.e., positive-valued nonlinear functions of the original queries and keys, which prove to be crucial for avoiding instabilities during training and provide more accurate approximation of the regular softmax attention mechanism.

Towards FAVOR: Fast Attention via Matrix Associativity
The decomposition described above allows one to store the implicit attention matrix with linear, rather than quadratic, memory complexity. One can also obtain a linear time attention mechanism using this decomposition. While the original attention mechanism multiplies the stored attention matrix with the value input to obtain the final result, after decomposing the attention matrix, one can rearrange matrix multiplications to approximate the result of the regular attention mechanism, without explicitly constructing the quadratic-sized attention matrix. This ultimately leads to FAVOR+.

Left: Standard attention module computation, where the final desired result is computed by performing a matrix multiplication with the attention matrix A and value tensor V. Right: By decoupling matrices Q′ and K′ used in lower rank decomposition of A and conducting matrix multiplications in the order indicated by dashed-boxes, we obtain a linear attention mechanism, never explicitly constructing A or its approximation.

The above analysis is relevant for so-called bidirectional attention, i.e., non-causal attention where there is no notion of past and future. For unidirectional (causal) attention, where tokens do not attend to other tokens appearing later in the input sequence, we slightly modify the approach to use prefix-sum computations, which only store running totals of matrix computations rather than storing an explicit lower-triangular regular attention matrix.

Left: Standard unidirectional attention requires masking the attention matrix to obtain its lower-triangular part. Right: Unbiased approximation on the LHS can be obtained via a prefix-sum mechanism, where the prefix-sum of the outer-products of random feature maps for keys and value vectors is built on the fly and left-multiplied by query random feature vector to obtain the new row in the resulting matrix.

Properties
We first benchmark the space- and time-complexity of the Performer and show that the attention speedups and memory reductions are empirically nearly optimal, i.e., very close to simply not using an attention mechanism at all in the model.

Bidirectional timing for the regular Transformer model in log-log plot with time (T) and length (L). Lines end at the limit of GPU memory. The black line (X) denotes the maximum possible memory compression and speedups when using a “dummy” attention block, which essentially bypasses attention calculations and demonstrates the maximum possible efficiency of the model. The Performer model is nearly able to reach this optimal performance in the attention component.

We further show that the Performer, using our unbiased softmax approximation, is backwards compatible with pretrained Transformer models after a bit of fine-tuning, which could potentially lower energy costs by improving inference speed, without having to fully retrain pre-existing models.

Using the One Billion Word Benchmark (LM1B) dataset, we transferred the original pre-trained Transformer weights to the Performer model, which produces an initial non-zero 0.07 accuracy (dotted orange line). Once fine-tuned however, the Performer quickly recovers accuracy in a small fraction of the original number of gradient steps.

Example Application: Protein Modeling
Proteins are large molecules with complex 3D structures and specific functions that are essential to life. Like words, proteins are specified as linear sequences where each character is one of 20 amino acid building blocks. Applying Transformers to large unlabeled corpora of protein sequences (e.g. UniRef) yields models that can be used to make accurate predictions about the folded, functional macromolecule. Performer-ReLU (which uses ReLU-based attention, an instance of generalized attention that is different from softmax) performs strongly at modeling protein sequence data, while Performer-Softmax matches the performance of the Transformer, as predicted by our theoretical results.

Performance at modeling protein sequences. Train = Dashed, Validation = Solid, Unidirectional = (U), Bidirectional = (B). We use the 36-layer model parameters from ProGen (2019) for all runs, each using a 16x16 TPU-v2. Batch sizes were maximized for each run, given the corresponding compute constraints.

Below we visualize a protein Performer model, trained using the ReLU-based approximate attention mechanism. Using the Performer to estimate similarity between amino acids recovers similar structure to well-known substitution matrices obtained by analyzing evolutionary substitution patterns across carefully curated sequence alignments. More generally, we find local and global attention patterns consistent with Transformer models trained on protein data. The dense attention approximation of the Performer has the potential to capture global interactions across multiple protein sequences. As a proof of concept, we train models on long concatenated protein sequences, which overloads the memory of a regular Transformer model, but not the Performer due to its space efficiency.

Left: Amino acid similarity matrix estimated from attention weights. The model recognizes highly similar amino acid pairs such as (D,E) and (F,Y), despite only having access to protein sequences without prior information about biochemistry. Center: Attention matrices from 4 layers (rows) and 3 selected heads (columns) for the BPT1_BOVIN protein, showing local and global attention patterns.
Performance on sequences up to length 8192 obtained by concatenating individual protein sequences. To fit into TPU memory, the Transformer’s size (number of layers and embedding dimensions) was reduced.

Conclusion
Our work contributes to the recent efforts on non-sparsity based methods and kernel-based interpretations of Transformers. Our method is interoperable with other techniques like reversible layers and we have even integrated FAVOR with the Reformer's code. We provide the links for the paper, Performer code, and the Protein Language Modeling code. We believe that our research opens up a brand new way of thinking about attention, Transformer architectures, and even kernel methods.

Acknowledgements
This work was performed by the core Performers designers Krzysztof Choromanski (Google Brain Team, Tech and Research Lead), Valerii Likhosherstov (University of Cambridge) and Xingyou Song (Google Brain Team), with contributions from David Dohan, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, and Adrian Weller. We give special thanks to the Applied Science Team for jointly leading the research effort on applying efficient Transformer architectures to protein sequence data.

We additionally wish to thank Joshua Meier, John Platt, and Tom Weingarten for many fruitful discussions on biological data and useful comments on this draft, along with Yi Tay and Mostafa Dehghani for discussions on comparing baselines. We further thank Nikita Kitaev and Wojciech Gajewski for multiple discussions on the Reformer, and Aurko Roy and Ashish Vaswani for multiple discussions on the Routing Transformer.


Google removes 3 Android apps for children, with 20M+ downloads between them, over data collection violations


When it comes to apps, Android leads the pack with nearly 3 million apps in its official Google Play store. The sheer volume also means that sometimes iffy apps slip through the cracks.

Researchers at the International Digital Accountability Council (IDAC), a non-profit watchdog based out of Boston, found that a trio of popular and seemingly innocent-looking apps aimed at younger users were recently found to be violating Google’s data collection policies, potentially accessing users’ Android ID and AAID (Android Advertising ID) numbers, with the data leakage potentially connected to the apps being built using SDKs from Unity, Umeng, and Appodeal.

Collectively, the apps had more than 20 million downloads between them.

The three apps in question — Princess Salon​, Number Coloring and ​Cats & Cosplay — have now been removed from the Google Play app store, as you can see in the links above. Google confirmed to us that it removed the apps after IDAC brought the violations to its attention.

“We can confirm that the apps referenced in the report were removed,” said a Google spokesperson. “Whenever we find an app that violates our policies, we take action.”

The violations point to a wider concern with the three publishers’ approach to adhering to data protection policies. “The practices we observed in our research raised serious concerns about data practices within these apps,” said IDAC president Quentin Palfrey.

The incident is being highlighted at a time when a lot of attention is being focused on Google and the size of its operation. Earlier this week, the US Department of Justice and 11 States sued the company, accusing it of monopolistic and anticompetitive behavior in search and search advertising.

To be clear, the app violations here are not related to search, but they underscore the scale of Google’s operation, and how even small oversights can lead to tens of millions of users being affected. They also serve as a reminder of the challenges of proactively policing individual violations on such a scale, and that those challenges can land in a particularly risky area: how minors use apps.

At least in the cases of two of the publishers, Creative APPS and Libii Tech (whose apps are built around the cast of characters illustrated at the top of this story), other apps are still live. And it also appears that versions of the apps are also still downloadable through APK sites (like this one). There are also versions on iOS (for example here), but Palfrey said it had not assessed iOS versions so it’s not clear if they are similarly leaking data.

The violation in this case is complex but is an example of one of the ways that users can unknowingly be tracked through apps.

Pointing to the behind-the-scenes activity and data processing that gets loaded into innocent-looking apps, IDAC highlighted three SDKs in particular used by the app developers: the Unity 3D and game engine, Umeng (an Alibaba-owned analytics provider known as the “Flurry of China” that some have described also as an adware provider), and Appodeal (another app monetization and analytics provider) — as the source of the issues.

Palfrey explained that the problem lies in how the data that the apps were able to access by way of the SDKs could be linked up with other kinds of data, such as geolocation information. “If AAID information is transmitted in tandem with a persistent identifier [such as Android ID] it’s possible for the protection measures that Google puts in place for privacy protection to be bridged,” he said.

IDAC did not specify the violations in all of the SDKs, but noted in one example that certain versions of Unity’s SDK were collecting both the user’s AAID and Android ID simultaneously, and that could have allowed developers “to bypass privacy controls and track users over time and across devices.”

IDAC describes the AAID as “the passport for aggregating all of the data about a user in one place.” It lets advertisers target ads to users based on signals for preferences that a user might have. The AAID can be reset by users. However, if an SDK is also providing a link to a users Android ID, which is a static number, it starts to create a “bridge” to identify and track a user.

Palfrey would not get too specific on whether it could determine how much data was actually drawn as a result of the violations that it identified, but Google said that it was continuing to work on partnerships and procedures to catch similar (intentional or otherwise) bad actors.

“One example of the work we are doing here is the Families ad certification program, which we announced in 2019),” said the spokesperson. “For apps that wish to serve ads in kids and families apps, we ask them to use only ad SDKs that have self-certified compliance with kids/families policies. We also require that apps that solely target children not contain any APIs or SDKs that are not approved for use in child-directed services.”

IDAC, which was launched in April 2020 as a spinoff of the Future of Privacy Forum, has also carried out investigations into data privacy violations on fertility apps and Covid-19 trackers, and earlier this week it also published findings on data leakage from an older version of Twitter’s MoPub SDK affecting millions of users.


Read Full Article

Leverage public data to improve content marketing outcomes


Recently I’ve seen people mention the difficulty of generating content that can garner massive attention and links. They suggest that maybe it’s better to focus on content without such potential that can earn just a few links but do it more consistently and at higher volumes.

In some cases, this can be good advice. But I’d like to argue that it is very possible to create content that can consistently generate high volumes of high-authority links. I’ve found in practice there is one truly scalable way to build high-authority links, and it’s predicated on two tactics coming together:

  1. Creating newsworthy content that’s of interest to major online publishers (newspapers, major blogs or large niche publishers).
  2. Pitching publishers in a way that breaks through the noise of their inbox so that they see your content.

How can you use new techniques to generate consistent and predictable content marketing wins?

The key is data.

Techniques for generating press with data-focused stories

It’s my strong opinion that there’s no shortcut to earning press mentions and that only truly new, newsworthy and interesting content can be successful. Hands down, the simplest way to predictably achieve this is through a data journalism approach.

One of the best ways you can create press-earning, data-focused content is by using existing data sets to tell a story.

There are tens of thousands — perhaps hundreds of thousands — of existing public datasets that anyone can leverage for telling new and impactful data-focused stories that can easily garner massive press and high levels of authoritative links.

The last five years or so have seen huge transparency initiatives from the government, NGOs and public companies making their data more available and accessible.

Additionally, FOIA requests are very commonplace, freeing even more data and making it publicly available for journalistic investigation and storytelling.

Because this data usually comes from the government or another authoritative source, pitching these stories to publishers is often easier because you don’t face the same hurdles regarding proving accuracy and authoritativeness.

Potential roadblocks

The accessibility of data provided by the government especially can vary. There are little to no data standards in place, and each federal and local government office has varying amounts of resources in making the data they do have easy to consume for outside parties.

The result is that each dataset often has its own issues and complexities. Some are very straightforward and available in clean and well-documented CSVs or other standard formats.

Unfortunately, others are often difficult to decode, clean, validate or even download, sometimes being trapped inside of difficult to parse PDFs, fragmented reports or within antiquated querying search tools that spit out awkward tables.

Deeper knowledge of web scraping and programmatic data cleaning and reformatting are often required to be able to accurately acquire and utilize many datasets.

Tools to use

TunesKit Spotify Music Converter: Best Spotify Music Downloader


If you used to be a user of Napster, you must be crazy about sharing music files with your friends. Back in the 2000s, people uploaded and downloaded songs on Napster. Nothing is better than owning the actual file of a song and being able to play it anywhere. But as online streaming keeps growing […]

The post TunesKit Spotify Music Converter: Best Spotify Music Downloader appeared first on ALL TECH BUZZ.