Data Cleaning Methods For Text Data – Nerd-out Wednesday Video
I discuss some tips, tricks, and methods that I learnt in cleaning text data. I clean my data in Stata, however in the past I have also done some of this in Excel and databases. You can also apply some of these ideas in things like Python.
Working with large public datasets is a pain, particularly if these large datasets have text data. This is quite useful for those that are looking at company names or text files.
Why do you want to clean your data? Having some data cleaning methods for text data is quite useful so that you can analyze the text data afterwards. What I usually do is use these data cleaning methods to then perform regressions on this text data.
I am not software specific for these data cleaning techniques in this video as they apply to all equally.
Here are some basic data cleaning methods that I have learnt with data cleaning of text data. Here are the data cleansing steps:
1. Save your data and make a copy. Only work with this copy.
2. Convert your text data to all caps or remove capitalization.
3. Remove all spaces in your data. There are usually functions like trim in Excel that can do this.
3. Create a for-loop to remove special characters. If you need to find the special characters, you can look them up in ASCII maps. Note, sometimes you have to look at special characters that are on Mac computers as well.
4. Look for common repeated words that do not add value to your data. For example, you can remove words like inc for incorporation. You can do this using some form of substr or find and replace. Careful though, you have to not remove these words when they matter.
5. Match data on string sequences. You can sometimes get rich information by using a duplicates function.
6. Fuzzy logic algorithms do not work as well as looking at matches using your own eyes for data cleaning techniques.
Check out:
How Do You Analyze Data In Research When Nothing Works? - Nerd-Out Wednesday
https://youtu.be/AqRFsv7umhk
Measurement error - Threats To Statistical Conclusion Validity - Nerd-out Wednesday
https://youtu.be/PNgF9nP9b8E
Range Restriction: Why It is Important For Statistical Conclusion Validity - Nerd-out Wednesday
https://youtu.be/tF-kp4GQJLk
*************************
David Maslach is a research professor of entrepreneurship, innovation, and business strategy, I discuss topics, such as behavioral science, strategy, innovation, and entrepreneurship, and apply these to my new peer proofreading and editing platform. Topics include the sharing economy, altruism, investing in technology, starting a business, and bounded rationality. My favorite videos pertain to incentives, goal setting, and learning from failure to drive behaviors such as weight loss, stopping telemarketers, creating novel technologies, and creating new movements.
https://r3ciprocity.com: Peer proofreading and editing platform
A new platform where you can earn credits by editing other people's documents. Use these credits to have your own work edited. If you do a good enough job, you can convert these credits to money.
The goal of the platform is to get people to 'pay it forward' and help other people out by creating incentives for people to give back.
Check out https://www.r3ciprocity.com
Please subscribe to the Youtube channel:
https://www.youtube.com/channel/UC5spxk7bNDMGPSHjW_8ndZA
About the Site 🌐
This site provides links to random videos hosted at YouTube, with the emphasis on random. 🎥
Origins of the Idea 🌱
The original idea for this site stemmed from the need to benchmark the popularity of a video against the general population of YouTube videos. 🧠
Challenges Faced 🤔
Obtaining a large sample of videos was crucial for accurate ranking, but YouTube lacks a direct method to gather random video IDs.
Even searching for random strings on YouTube doesn't yield truly random results, complicating the process further. 🔍
Creating Truly Random Links 🛠️
The YouTube API offers additional functions enabling the discovery of more random videos. Through inventive techniques and a touch of space-time manipulation, we've achieved a process yielding nearly 100% random links to YouTube videos.
About YouTube 📺
YouTube, an American video-sharing website based in San Bruno, California, offers a diverse range of user-generated and corporate media content. 🌟
Content and Users 🎵
Users can upload, view, rate, share, and comment on videos, with content spanning video clips, music videos, live streams, and more.
While most content is uploaded by individuals, media corporations like CBS and the BBC also contribute. Unregistered users can watch videos, while registered users enjoy additional privileges such as uploading unlimited videos and adding comments.
Monetization and Impact 🤑
YouTube and creators earn revenue through Google AdSense, with most videos free to view. Premium channels and subscription services like YouTube Music and YouTube Premium offer ad-free streaming.
As of February 2017, over 400 hours of content were uploaded to YouTube every minute, with the site ranking as the second-most popular globally. By May 2019, this figure exceeded 500 hours per minute. 📈
List of ours generators⚡
Random YouTube Videos Generator
Random Film and Animation Video Generator
Random Autos and Vehicles Video Generator
Random Pets and Animals Video Generator
Random Travel and Events Video Generator
Random People and Blogs Video Generator
Random Entertainment Video Generator
Random News and Politics Video Generator
Random Howto and Style Video Generator
Random Education Video Generator