NLP with SpaCy, Dataflow ML and BigQuery ML Clustering

This blog explores the implementation of text clustering using SpaCy and the machine learning capabilities of Google Cloud Platform (GCP). The approach involves utilizing Conversational AI, which leverages advanced Natural Language Processing (NLP) models trained on a massive dataset.

The blog discusses the architecture roadmap, where 9311 sentences are segregated and placed in a BigQuery table through a CSV upload.

SpaCy generates document vectors for each row in the pipeline. The vectors are then clustered using BigQuery's ML powers. The text clustering mechanism can be applied to document plagiarism detection, robust question-answering systems, and other applications.

The fuzzier approach may be adopted to increase accuracy. The pipeline and clustering metrics can be viewed in the BigQuery console and Looker Studio.