Web Scraping Reddit with Google Scripts

I am working on a little side project that involves mining Reddit data. It fetches a listing of all posts on different subreddits and copies the obtained data to a Google spreadsheet for further analysis (more on the project later).

Reddit, unlike most websites, allows web scraping as long as the crawler scripts make no more than one request every two seconds to the Reddit servers (see rules). You don’t even need a developer account or an API key to perform scraping on Reddit.

There are popular tools like wget, Site Sucker (Mac) or HTTrack Website Copier (Windows) that can download entire websites for offline use but they are mostly useless for scraping Reddit data since the site doesn’t use page numbers and content of pages is constantly changing. A post maybe listed on the first page of a subreddit but it could find itself on the third page the next second as other posts are voted to the top.

While there exists PHP and Python libraries for scraping Reddit, they are too complicated for the non-techies. Fortunately, there’s always Google Apps Script to the rescue. Here’s what you can do to pull data from any Subreddit on Reddit automatically.

1. Open the Google Sheet and choose File – Make a copy to copy this sheet in your Google Drive.


2. Go to Tools -> Script editor and copy-paste the Reddit Scraper Script. You can change “LifeProTips” to any other subreddit name.


3. While in the script editor, choose Run -> Run and authorize the script.

That’s it. The script will run in the background automatically pulling content from Reddit into the Google spreadsheet. And it stops automatically once all the posts* of that Reddit have been fetched.

[*] All Subreddits on Reddit display a maximum of 1000 posts – you can’t go beyond that number even while manually browsing a subreddit.

---

This story, Web Scraping Reddit with Google Scripts, was originally published at Digital Inspiration on 09/01/2014 under Javascript, Internet

6 Life Habits That Programming Could Teach You Today

Everything important that you need to know about living a successful life, you can get from a computer program. Don’t believe me? Read on. When I first started programming as a young kid, it only amounted to copying foreign-looking words and symbols out of a computer magazine so that my brother and I could get our old Franklin 64 with a dual floppy drive to play a cute little digital tune at us. Back then, there weren’t many life lessons to discern out of that cryptic text. A number of years later, in high school Pascal class, things started to...

Read the full article: 6 Life Habits That Programming Could Teach You Today

Interesting Websites to Visit When You’re Bored

I have added these non-tech related websites in my “weekly” bookmarks folder. I don’t visit them every single day but on weekends or when I am feeling bored and they never disappoint. Maybe you should check them out too.

The 15 Interesting Websites

1. Dear Photograph – The website features photographs of people holding up old pictures that were taken at the exact spot several years ago. Some of the best photographs are now compiled in a book (dearphotograph.me).


2. Textastrophe – This website is a reminder of why you shouldn’t post your cell phone number on public websites. The creator starts a text conversation with random numbers found on Craigslist ads and publishes the transcript on the internet. The chats are funny and I hope its real because screenshots can be faked too (textastrophe.com).


3. Stratocam – This is like an online self-running slideshow showcasing some of the best satellite images found on Google Maps. Run the website in full-screen mode (Ctrl+Shift+F) and you may discover pretty and unexpected place on our beautiful earth (stratocam.com).


4. Awkward Family Photos – This is a place where people share embarrassing pictures of their loved ones in weird settings that are both funny and entertaining. The website has spawned several coffee-table books as well (awkwardfamilyphotos.com).


5. Worldcam – This website pulls geo-tagged pictures posted on Instagram for Foursquare venues so you know what’s happening right now at your favorite place. For instance, here’s the livestream of a mall in Thailand which also happens to be the most Instagrammed place in the world (worldc.am).


6. This Is Why I’m Broke – Unique products on the Internet that you never knew even existed. Stuff like an iPhone case that doubles as a bottle opener, sandals that resemble computer keyboards or a boat from World War II that you can actually buy (thisiswhyimbroke.com).


7. Is It Normal – Do you like to dip French fries in ice cream? The website will help you understand whether your thoughts, feelings and urges are normal or weird and just unique to you. People can ask questions and once it gets approved, the community can weigh in their thoughts.You can also ask embarrassing questions here (isitnormal.com).
   

   Also see: The Most Useful Websites

   
   


8. Lamebook – Weird and hilarious stuff that people post on Facebook. The names and profile pictures of the Facebook users are blurred to protect real identity (lamebook.com).


9. Honest Slogans – Created by graphic designer Clif Dickens, Honest Slogans pokes fun at iconic brands by remixing their logos with slogans that convey what people actually think about that brand. Like the one for Men’s Health magazine that says – “Every issue promises perfect abs” (honestslogans.com).


10. Rich Kids of Instagram – The social sites are the new hangouts for people to show off and this tumblr is a perfect example. It captures the lavish lifestyle of the rich through pictures posted on Instagram (richkidsofinstagram.tumblr.com).


11. Timelapse – Watch how a place has changed over a period of three decades through a series of satellite images from Google Maps. You can use the “Explore our World” link to watch the timelapse of any location on the planet (world.time.com).


12. Reasons My Son Is Crying – If you are a new parent, you’ll love this one. It documents the different reasons that little children cry for as submitted by parents around the world. Also available as a book (reasonsmysoniscrying.com).


13. LOL My Thesis – This site created by an Harvard University senior creates funny and lighthearted one-line summaries of thesis like this one submitted by a Sociology student – “It turns out, an only child never fights with its siblings.” (lolmythesis.com).


14. Larson+Shindelman – They pick a few geo-tagged tweets from Twitter, travel to the physical spot from where that tweet was sent and then take a picture. The original tweet is used as a caption for the photograph and the results are often incredibly touching. Like that photo outside the hospital with the caption “Amy is dying” (larson-shindelman.com).


15. 9-Eyes – A collection of interesting images as captured by Google Street view cards around the world. Also check out mapcrunch.com that catalogues street view scenes by location.

---

This story, Interesting Websites to Visit When You’re Bored, was originally published at Digital Inspiration on 09/01/2014 under Websites, Internet

5 Security Tips To Consider When Using a Microsoft Account

Microsoft wants Windows 8 users to log into their computers with a Microsoft account, not a standard old local user account. You can’t use much of the new user interface without a Microsoft account — you can’t even upgrade to Windows 8.1 without one. Along with this new focus on Microsoft accounts comes new security concerns. The account you use to log into your computer is now an online account and you need to worry about securing it. There are advantages to using a Microsoft account, as it allows you to sync your settings, files, apps, and other data between...

Read the full article: 5 Security Tips To Consider When Using a Microsoft Account

Contract Freedom, Modular Smartphones, DuckDuckGrow, Kanye Vs. Coinye [Tech News Digest]

Today in Tech News Digest, T-Mobile offers you money to switch contracts, ZTE teases a modular smartphone, DuckDuckGo shows off, Dell brings 4K monitors to the mainstream, biased reviews of Yahoo’s app are questioned, Kindle for iOS is improved, and Kanye West goes after cryptocurrency Coinye. T-Mobile Offers Contract Freedom T-Mobile is offering to pay the costs associated with terminating your mobile contract early. The company wants you to switch to them, and in an effort to persuade you it’s worth doing so, will pay your early termination fees (of up to $350 per line) with one of the three...

Read the full article: Contract Freedom, Modular Smartphones, DuckDuckGrow, Kanye Vs. Coinye [Tech News Digest]

Ultimate Windows Tweaker 3.0 Released For Windows 8.1

CLICK HERE TO SEE FULL POST



Those of you following IntoWindows or any other popular tech website ever since the release of Windows 7 probably have noticed that most of the tweaking applications that were released for Windows 7 haven’t been updated to support Windows 8 or Windows 8.1. For those who have started using Windows or computers recently, hundreds of […]

How Do You Obtain Music? [We Ask You]

Music is a universal language that binds all humans together in a very real way. While the specific styles and genres of music listened to by different cultures vary wildly, they all share common ground. In other words, music is a very powerful force. It’s been true for a long time: the music industry is in a state of flux. The Internet having upset the apple cart that had been trundling merrily on for several decades. Now, each of us has multiple options open to us for getting our music fix. This multitude of options forms the basis for this...

Read the full article: How Do You Obtain Music? [We Ask You]

8 Easy Ways To Get Started With A Photography Habit Today

You don’t have to wait for the turn of a new year to start a new habit. Each day can be a new beginning. A hobby like photography also needs the discipline of a habit, and you can give it an extra motivational push with the help of the Web. You shouldn’t let your camera collect dust – think differently and inspiring photography ideas will come to you. If all fails, the Web and the number of popular photography blogs offer a constant stream of ideas. So, are you stuck in a rut? Is your camera cocooned snugly in its...

Read the full article: 8 Easy Ways To Get Started With A Photography Habit Today

How To Turn Random Snapshots Into A Uniform-Looking Photo Series

A few tricks can save you hours of editing in Photoshop. Let’s say you have a random collection of photos. You or whoever the photographer was didn’t have the foresight to take them with the same camera or lens, but now you wish you could combine them and make them look like they were created during a single photo shoot. Rather than tediously editing each photo, you could run all of them through the same preset filter to create that desired uniform look. Let me explain… Pick Your Photo Collection Maybe you have a collection of family photos, taken by...

Read the full article: How To Turn Random Snapshots Into A Uniform-Looking Photo Series

Technophilia 95 Live From MakeUseOf

We’re reunited and it feels…okay. This week: backdoors guests are best! Infrequent guests knock your front door, which you’d otherwise never open. Good friends know to use the backdoor, and by this standard the NSA is apparently a very, very good friend to everyone. McAfee anti-virus is changing its name to stop being associated with John McAfee, the drug-crazed alleged killer who founded the company. They should also consider changing the software. Like, completely. Or just get rid of it. Forever. Also this week: the Cat In The Hat becomes public domain this year! Just kidding, nothing will ever become...

Read the full article: Technophilia 95 Live From MakeUseOf

Create PC Settings Shortcut On Desktop In Windows 8.1

CLICK HERE TO SEE FULL POST



The updated PC settings, the modern UI Control Panel designed especially for touch devices, in Windows 8.1 includes settings for lock screen, SkyDrive, sync, share, search, user accounts, network, recovery and device settings. With default settings, one can launch PC settings by clicking Settings on the Charms bar and then clicking Change PC settings in […]

Your Top Tech Resolutions For 2014 [We Ask You Results]

We’re now well into 2014, with 2013 (and its memes) being a dim and distant memory for most of us. Tradition dictates that a new year brings with it an opportunity for us all to improve ourselves, as we strive to be better people this year than last year. Which is where new year’s resolutions come into play. These vows to stick to something for a whole year or improve at something over the course of the year are a fun way in which we can all pinpoint the areas of our lives we’re currently unhappy with. And your new...

Read the full article: Your Top Tech Resolutions For 2014 [We Ask You Results]

How To Disable Forced Restarts After A Windows Update

It’s happened to all Windows users. You’re using your computer and Windows decides you need to reboot, hassling you with pop-ups that continue to bug you all day long. If you step away from your computer and miss the pop-up, Windows will automatically reboot your computer. You may come back to your computer and find that all your open programs are gone, as Windows decided to reboot without your permission. This can be maddening. There’s a good reason for rebooting after updates, as rebooting ensures that security updates actually take effect. But Microsoft has gone too far — they shouldn’t...

Read the full article: How To Disable Forced Restarts After A Windows Update

How I Nearly Got Conned Via A Western Union Transfer Scam

Here’s a little story about the latest “Nigerian scam“, which is all too obvious in hindsight and yet so believable when you’re on the hook. Now, I’m a regular online shopper on Amazon, eBay, and the like. I know the sorts of scams you see on Craigslist and the very similar eBay scams you’re likely to come across. I know basic rules of not getting fleeced in marketplace transactions and most of the likely online scams. Yet, still, this one almost got me. The Scam It all started on Facebook. A woman posted to a local buy-sell group with a...

Read the full article: How I Nearly Got Conned Via A Western Union Transfer Scam

Forget Kickstarter: How To Crowdfund From Your Own Website

Crowdfunding has taken off in a big way in the past few years. It’s easy to see why, as the Internet offers a way for many likeminded people to club together. But you don’t need to use a service like Kickstarter if you’re already running your own popular website… Run Crowdfunding Campaigns On Your Own Website It should come as no surprise that MakeUseOf runs regular crowdfunding articles. The ease with which it is possible to back interesting new projects – the majority of which would never get off the ground a few years ago – is exciting for many,...

Read the full article: Forget Kickstarter: How To Crowdfund From Your Own Website

Pebble Steel, Toshiba Chromebook, Yahoo Malware, Michael Bay [Tech News Digest]

Today in Tech News Digest, new smartwatch hardware is announced in the form of Pebble Steel, Toshiba launches its first Chromebook, AT&T potentially kills net neutrality with Sponsored Data, Yahoo users are warned of a malware threat, Panasonic announces Firefox OS-powered smart TVs, retro games are made playable online thanks to the Console Living Room, and Michael Bay fails to deliver at CES 2014. Pebble Steel Announced CES 2014 has kicked off, and new hardware is being shown off left, right, and center. The biggest reveal on day one was arguably the new Pebble Steel, which turned heads thanks to...

Read the full article: Pebble Steel, Toshiba Chromebook, Yahoo Malware, Michael Bay [Tech News Digest]

Learn To Draw Anything Thanks To YouTube [Stuff to Watch]

YouTube is a fantastic learning tool, and that’s primarily because many of us learn best by watching someone do something. This rings especially true when it comes to drawing – if you’re bursting with creativity but are falling short in the artistic department, YouTube might be able to help. Drawing is a skill best nurtured by practice, but a little help can go a long way. Whether you’re having trouble drawing facial features like eyes or haven’t got a clue how to tackle a 3D perspective, it’s YouTube to the rescue! Drawing With Karl Gude One of my favourite instructional...

Read the full article: Learn To Draw Anything Thanks To YouTube [Stuff to Watch]

9GAG: Laugh Until You Can’t Laugh Anymore

Maybe you’ve heard of this meme-sharing site which shares some similarities with Reddit (which is a really cool website), or maybe this is your first encounter with 9GAG. Either way, you’re bound to enjoy the many user submitted memes that make it to this site. Essentially, 9GAG is just a collection of memes and images that anyone can upload. Most of them tend to be funny, some are brutally honest (see: confession bear, unpopular opinion puffin), some are cute, and some are just fascinating. It’s completely free and has free Android and iOS apps as well. Since your experience will...

Read the full article: 9GAG: Laugh Until You Can’t Laugh Anymore

Give Your Text Stunning Designs Easily With Notegraphy

Is this text boring? And this? And this? I’m not talking about the actual content. I’m referring to the way it looks. Text can be boring or interesting, depending on what it actually says, but no matter where you read it — on a website, in a book, or even on a social network — it’s likely to look about the same…in spite of different fonts. Personally, this never bothered me before, but that was before I tried Notegraphy. Notegraphy makes it easy to create stylized snippets of text for yourself, or to share on social networks, and can also...

Read the full article: Give Your Text Stunning Designs Easily With Notegraphy

6 Ways To Make Nature Sounds A Part Of Your Day

A dose of nature a day will keep the doctor away. Several scientific studies have shown that nature sounds can significantly decrease anxiety, stress, and pain. Next time you’re bursting from stress or have a roaring headache, try relaxing to the sounds of nature before automatically popping a pill. In a previous article, I showed you how to bring nature to your desktop. Today, I’ll point out how to get the sounds that will benefit your health and well-being. Take A Walk Outside Nature restores mental functioning in the same way that food and water restore bodies. - Adam Alter,...

Read the full article: 6 Ways To Make Nature Sounds A Part Of Your Day