This book is intended as a resource for people who are interested in using computers to help process natural language. A natural language refers to any language spoken by humans, either currently (e.g., English, Chinese, Spanish) or in the past (e.g., Latin, ancient Greek, Sanskrit). Annotation refers to the process of adding metadata informa tion to the text in order to augment a computer’s capability to perform Natural Language Processing (NLP). In particular, we examine how information can be added to natural language text through annotation in order to increase the performance of machine learning algorithms—computer programs designed to extrapolate rules from the infor mation provided over texts in order to apply those rules to unannotated texts later on.

Create your own natural language training corpus for machine learning. Whether you’re working with English, Chinese, or any other natural language, this hands-on book guides you through a proven annotation development cycle—the process of adding metadata to your training corpus to help ML algorithms work more efficiently. You don’t need any programming or linguistics experience to get started.

Using detailed examples at every step, you’ll learn how the MATTER Annotation Development Process helps you Model, Annotate, Train, Test, Evaluate, and Revise your training corpus. You also get a complete walkthrough of a real-world annotation project.

• Define a clear annotation goal before collecting your dataset (corpus)
• Learn tools for analyzing the linguistic content of your corpus
• Build a model and specification for your annotation project
• Examine the different annotation formats, from basic XML to the Linguistic Annotation Framework
• Create a gold standard corpus that can be used to train and test ML algorithms
• Select the ML algorithms that will process your annotated data
• Evaluate the test results and revise your annotation task
• Learn how to use lightweight software for annotating texts and adjudicating the annotations

This book is a perfect companion to O’Reilly’s Natural Language Processing with Python.