Introducing Spark NLP: State-of-the-art natural language processing at scale

Natural language processing is a key component in many data science systems that must understand or reason about text. Common use cases include question answering, paraphrasing or summarization, sentiment analysis, natural language BI, language modeling, and disambiguation. Building such systems usually requires combining three types of software libraries: NLP annotation frameworks, machine learning frameworks, and deep learning frameworks.

David Talby offers an overview of the NLP library for Apache Spark, which natively extends Spark ML’s pipeline APIs, enabling zero-copy, distributed, combined NLP and ML pipelines that leverage all of Spark’s built-in optimizations. The library implements core NLP algorithms, including lemmatization, part-of-speech tagging, dependency parsing, named-entity recognition, spell checking, and sentiment detection. David then demonstrates how to use these algorithms to build commonly used pipelines, using PySpark on notebooks that will be made publicly available after the talk.