AI/ML Engineer

Job Title - AI/ML Engineer

Experience - 6+ years

Location - Hyderabad, Vizag

Primary Skills - Python, PyTorch, TensorFlow, LLM Models

Roles and Responsibilities:

1. Data Collection, Preprocessing, and Management Data Collection: Gather diverse datasets for training LLMs or other machine learning models. This may include text data from various sources like web scraping, databases, or APIs. Data Preprocessing: Clean and preprocess raw text data (e.g., tokenization, stemming, lemmatization, removing stopwords) for NLP tasks using tools such as SpaCy, NLTK, or custom preprocessing pipelines. Data Augmentation: Create synthetic data or perform augmentation techniques to enrich training datasets, particularly in scenarios where large labeled datasets are scarce. Data Pipeline Development: Build automated and scalable data pipelines for continuous data ingestion, cleaning, and feeding into models.
2. Building and Training Machine Learning Models Model Selection and Design: Design and implement deep learning architectures for specific use cases (e.g., LLMs for NLP tasks like sentiment analysis, text summarization, question answering). Model Development Using PyTorch and TensorFlow: PyTorch: Build and train custom neural networks using PyTorch, leveraging its dynamic computation graph and flexibility for research and experimentation. TensorFlow: Implement scalable, production-ready models using TensorFlow (including TensorFlow Hub and Keras for high-level model building). Training Large Models: Train large models like transformers (e.g., BERT, GPT, T5) using large-scale datasets. Efficiently handle high computational requirements for these models, potentially using cloud services (AWS, GCP) or GPUs. Fine-Tuning Pre-trained Models: Fine-tune pre-trained models like BERT, GPT-3, or other LLMs on task-specific data to improve performance on downstream applications. Model Evaluation: Use evaluation metrics like accuracy, F1 score, BLEU score (for text generation), or perplexity to assess model performance. Perform cross-validation and hyperparameter optimization.
3. Model Optimization and Scaling Hyperparameter Tuning: Experiment with hyperparameters (e.g., learning rates, batch sizes, number of layers, dropout rates) to enhance model performance and prevent overfitting. Optimization: Use model optimization techniques such as quantization, pruning, and knowledge distillation to reduce the size and improve the inference speed of large models. Distributed Training: Implement distributed training using PyTorch Distributed or TensorFlow’s MirroredStrategy to train large models efficiently across multiple GPUs/TPUs.
4. Model Deployment and Integration Model Deployment: Deploy AI/ML models into production environments (e.g., AWS SageMaker, Google AI Platform) ensuring scalability, security, and robustness. API Development: Build APIs or microservices for serving models, enabling real-time predictions or batch processing using frameworks like Flask, FastAPI, or TensorFlow Serving. Model Monitoring: Implement monitoring systems to track the performance and accuracy of models in production. Detect model drift or degradation over time and retrain when necessary. Scalability and Optimization: Ensure that the models can scale to handle large-scale inference workloads. Use TensorFlow Lite for edge devices or ONNX for cross-framework deployment.
5. Research and Experimentation Cutting-Edge Research: Stay up to date with the latest advancements in machine learning, especially in transformer models and NLP, and incorporate state-of-the-art techniques into your work. Innovation: Experiment with novel approaches for improving model accuracy, efficiency, or generalization (e.g., new transformer variants, unsupervised pretraining techniques). Contributing to Open Source: Contribute to or develop open-source projects that enhance machine learning tools, especially in the field of NLP and LLMs.