AI has gone mainstream, and there’s no turning back. The next generation would probably gawk at the world that once existed without AI. So, where did it all begin?
You wouldn’t be surprised if I claim that a simple chat interface helped us teleport into the AI-first world. While AI has been on the scene for decades, it was ChatGPT that propelled it into everyday conversations, making AI accessible and perhaps indispensable.
People may have initially mistaken ChatGPT for a regular rule-based chatbot (that you find on websites) that often failed to provide meaningful answers. But ChatGPT was different. It gave you contextual answers precise ones at that, reducing our need to sift through endless Google search results for answers. More importantly, it delivered unprecedented value.
It was disruptive innovation at its best. But what made this possible? Those in DataOps or MLOps would point to the 4Vs – Volume, Velocity, Variety, and Veracity. In fact, they would declare these four pillars determine the success of any ML model, and ChatGPT was no exception. And behind these 4Vs stand the 4Ps Data Annotators, Data Engineers, Data Scientists, and DevOps Engineers: the key personas who ensure AI models function at peak efficiency. Without a strong DataOps foundation, even the most advanced AI models would struggle to process vast amounts of data efficiently.
Let’s Slice and Dice the 4Vs of DataOps
The Geminis and Groks of today are trained on billions of text samples (websites, books, research papers, and code repositories). They demonstrate the sheer volume at play. Their ability to process millions of queries in seconds showcases velocity. And the variety of training data makes them truly multimodal: it can interpret an abstract painting as well as summarize complex legal documents. Most importantly, the veracity (accuracy and correctness) of their outputs is what drives widespread adoption. Put together, these 4Vs of DataOps create unparalleled value.
For businesses building prediction engines, recommendation systems or fraud detectors, success depends on the AI model’s ability to uphold these 4Vs. The 4Ps catalyze this process, ensuring precise and impactful outcomes.
How the 4Ps Drive the 4Vs of DataOps in Your Custom Enterprise ML Models
The 4Ps are the four essential data personas who play a pivotal role in your DataOps cycle to ensure ML models accurately predict patterns and deliver intended outputs. They are: Data Annotators, Data Engineers, Data Scientists, and DevOpsEngineers.
Let’s quickly analyze their roles and how they contribute to providing efficient data engineering services for ML model deployment..
Data Annotators (D1)
Consider this example. A freight carrier that moves shipments between places using trucks, ships, or planes implements an ML-based price predictor to estimate the costs for each shipment. Freight customers will use this price prediction engine to gauge pricing, which is influenced by variables such as time, day/season, rates (spot or contract), shipping lane, and cargo type.
Let’s imagine a full truckload (FTL) shipment of frozen meat that needs to be moved in 24 hours from Dallas to Chicago in December.
The ML model predicts the price based on:
- Time: Expedited time incurs higher costs
- Season: December peak shipping season drives up costs
- Rate: Spot rates exceed standard rates
- Lane demand: Dallas to Chicago is a high-demand lane
- Freight type: Frozen meat is temperature sensitive cargo that requires Reefer vans, in turn affecting pricing.
However, before an ML model can make these predictions, the ML model must be trained on properly labeled historical data. This is where data annotators come into play. Data annotators categorize and annotate historical information to train the ML models effectively. Figure 1 shows how shipments are typically annotated:
| Variable | Entity | Label |
|---|---|---|
| Shipment type | Frozen meat | Temperature sensitive |
| Vehicle type | Reefer van | Consumes extra fuel |
| Source to destination | Dallas to Chicago | High demand route |
| Month of travel | December | Peak season |
| Time | 24 hours | Peak transit |
D2 (Data Engineers)
Data annotators need structured datasets with accurate information to label them meticulously. This falls under the purview of data engineers who write algorithms to fetch data from multiple sources. In the freight example we are considering, data engineers could collect historical data from a TMS system by querying structured databases such as Postgresql and Snowflake. Such robust data engineering services ensure efficient integration to enable high-quality datasets for accurate model training.
Data collection is followed by cleansing, where data engineers add the missing information, remove erroneous or duplicate information, and structure the data before storing them in a datalake or lakehouse architecture. Data annotators then query these datasets, label them, and store the labelled data back in the data repository.
D3 (Data scientists)
The ML model is ready to be trained with labeled data sets. But for it to predict a price, it has to know what variables are at play. Through a process known as feature engineering, the ML model is exposed to highly relevant variables that enhance its accuracy.
Three prominent processes play a crucial role:
1Feature addition: In the above pricing predictor example, if we introduce weather impact as a new variable, it enhances the context for the ML model. Firstly, bad weather would have slowed down transit. Secondly, since you were carrying meat, which is temperature-sensitive, the reefer van would have consumed more fuel to maintain the temperature. All these have an ultimate bearing on the end-price.
| Variable | Entity | Label |
|---|---|---|
| Weather impact | Dec 15 (snowstorm) | Adverse weather |
Thus, through feature addition, new variables supply more context to ML models that in turn enhances its prediction accuracy.
2Feature extraction: The pricing model is not dependent on variables such as shipment ID or broker fees. Feature extraction ensures only the relevant variables go into the ML model for training.
3Feature transformation: ML models do not understand numerical or categorical variables. How will they understand if you say, the spot rate of $3.8 per mile is considerably higher? It has to be scaled. For instance, after scaling, you will say a $3.8 spot rate is 0.75 on a scale of 0 to 1. They also don’t understand categorical variables such as freight type, shipment type, and source to destination. These will have to be encoded like the one below.
| Variable | Value | Label | Encoded value |
|---|---|---|---|
| Shipment Type | Frozen Meat | Temperature Sensitive | Temperature sensitive = 1 Non-sensitive = 0 |
| Vehicle type | Reefer van | Consumes extra fuel | Reefer van = 1 Flatbed = 0 |
| Source to destination | Dallas to Chicago | High demand route | High demand lane = 1 Low demand lane = 0 |
| Month of travel | December | Peak season | Peak Season = 1 Off season = 0 |
| Time | 48 hours | standard transit | Peak transit = 1 Standard transit = 0 |
Once the feature engineering is complete, the model uses a suitable algorithm that takes into account the multiple variables and predicts an accurate pricing for a shipment.
Data scientists may use an ML learning method called random forest. It’s an industry standard that works well with multiple variables, handles missing data efficiently, and provides a transparent pricing engine. This helps even shippers to understand the logic behind the costing.
D4 (DevOps Engineer)
At this stage, the DataOps cycle has given us a trained model ready to be deployed. Now, the DevOps engineers set up the necessary infrastructure to deploy the model. They ensure the DataOps model is available as an API service, take care of automating the deployment of each and every version of the ML model, and continuously monitor the performance of the model.
The following are the primary functions of the DevOps engineer in the context of the freight pricing example:
Package the price predictor program in a docker container, and ensure the model works perfectly fine across all environments (dev, staging, and production).
- Make the model available through a Rest API for shippers or 3PLs to access the service.
- Use GitHub and Kubernetes services to ensure every model version is automatically deployed and hence only the latest version is always available.
- Constantly monitor the model performance, which has a tendency to degrade over time. Continuously retrain to minimize model drift and enforce data governance to maintain the quality and integrity of data used for retraining.
- Make sure the model scales up whenever there are more users trying to access the pricing service.
Now, how do these 4Ps or four key personas contribute to the 4Vs of DataOps ? In our freight example, each one of them had a role to play to ensure every V(value) met the required benchmarks.
| DataOps Personas | Volume | Velocity | Variety | Veracity |
|---|---|---|---|---|
| Data Engineer | Manages large-scale data ingestion from multiple sources | Builds data pipelines for real-time data processing | Integrates structured and unstructured data | Cleans data to ensure accuracy and reliability |
| Data annotator | Labels data with meaningful categories | Validates and corrects labels to maintain data quality | ||
| Data Scientist | Designs models that can process high-velocity data streams efficiently | Enriches the ML model’s predictive power by adding new data variables | Improves data reliability through feature engineering | |
| DevOps Engineer | Implements auto-scaling infrastructure to handle high API request volumes | Sets up CI/CD pipelines to quickly deploy model updates | Monitors model performance and data pipeline health to ensure accurate predictions |
Though these four DataOps personas contribute to the success of the ML models, there are other roles that assume significance in successful deployments. For example, it is the ML engineers who bridge the gap between model development and production use. The business analysts act as domain experts, ensuring the model works to solve the intended business challenge. The data architect sets up the foundation for a robust data infrastructure.
And let’s remember that DataOps, much like DevOps, is an ongoing effort. People in varied roles come together to leverage data engineering services and advanced tools within a structured framework, to leverage maximum value from raw business data.
Accelerating ML deployments
At Trigent, a leader in Gen AI, our Ninja teams of data engineers, analysts, and scientists work together to create efficient ML models that align with and often exceed the established benchmarks in Volume, Velocity, Variety, and Veracity. For companies with AI ambitions, we help them translate their idea into an MVP in six weeks.
The AI launchpad gets you priority access to our Data Engineers & ML Specialists, a team of full stack Ninja Developers for GenAI application development, and the AI Studio for CX design. It empowers you to fully understand the AI potential in your domain, and craft a custom approach to suit your business requirements.
Leverage our DataOps and data engineering services to build efficient pipelines and deploy scalable AI solutions faster.
Unlock Your ML Power with Trigent’s AI launchpad
