- By: Áine Byrne
- Published on:
Share on:
Feature Engineering and the Art of Learning
Academic Log: Stardate 2025.0930. In our previous logs, we jammed with Machine Learning (ML) fundamentals through music and charted the warp-speed rise of foundation models, Starfleet’s elite, trained on galaxy-scale data and ready for any mission. But even the most experienced Captains began as Cadets, sitting in classrooms, buried in textbooks, running simulations, and wrestling with homework. Before Warp Drives and Holodecks, there was the ML Academy.
This time, we return to the ML Academy to understand how models learn from the ground up. From defining the mission to passing the final exam, we’ll explore deeper the essential stages of machine learning, each one a lesson in how models learn, adapt, and graduate from raw data to intelligent predictions.
If we beam ourselves back to the Music Log: Stardate 2025.0829, we explored the five steps to machine learning mastery (Machine Learning Pipeline). Whether you’re building a recommendation engine or forecasting market trends, the journey always begins with one essential: A question, task or problem to be solved. So, grab your dataPADD and report directly to the ML Academy, as class is in session.
Click on each step below, to learn more.
1. Mission Briefing: What do you want the system to learn?
Every ML student begins with the following mission: What do you want the system to learn? Is it diagnosing medical conditions, predicting market trends, or translating Klingon opera? This is your task definition, it’s the first and most critical step in the machine learning pipeline.
2. Fuel and Format: Gathering the right data and representing it well
But a mission without data is like a starship without fuel. Next, step 2, you need to gather data. But not just any data, you need the right kind, and just as importantly step 3, you will need to know how to represent it. Remember that nerdy term from our Music Log: Stardate 2025.0829 and how to turn raw data into something a model can understand? The representational challenge.
3. Represent the data
Before the age of end-to-end deep learning, we tackled this challenge manually, through feature engineering. That meant distilling raw data into structured formats the model could understand. Think back to the Music Log: Stardate 2025.0829 when we filled out that jazzy spready (or musical questionnaire) for a song. Whether it’s a music track or a patient record, we extracted relevant attributes and slotted them into a feature vector.
The process of deciding what matters and how to encode it is the art of feature engineering, and it’s where traditional ML methods truly shine. This art was the backbone of learning, however it came with a catch, but more on that later on in the log.
4. Choose Your Architecture: The learning blueprint
With your task defined, data gathered, and representation chosen, it’s time to choose the model architecture. This is step 4 in the machine learning pipeline, the blueprint that determines how your model will learn.
Just like choosing the right starship for a mission, your architecture shapes the model’s capabilities. Will it be a simple logistic regression runabout or a deep neural network cruiser? The choice depends on the complexity of the task, the nature of the data, and the resources available.
Once the architecture is chosen, your model is ready to begin training. And just like a Cadet at Starfleet Academy, it needs structured practice to perform well in the field.
Hyperparameters: The Bridge Controls
In our Captain’s Log: Stardate 2025.0911 we described hyperparameters as the dials and switches that set the course and guide the learning process. Learning rate, batch size, model architecture, these settings shape how the model trains, adapts and performs.
Set them too aggressively, and your model might overshoot, memorising the training data instead of learning from it. Set them too gingerly, and it may never reach escape velocity. When overfitting strikes (memorising instead of generalising), it’s time to recalibrate.
You iterate. You optimise. You explore new configurations, tweaking the controls until your model performs not just on familiar data, but on the unknown. Like a seasoned officer adjusting course mid-flight, you guide the model toward generalisation, ensuring it’s ready for any mission Starfleet throws its way.
5. Validate Your Solution: The academy exam
Training alone isn’t enough. To ensure your model isn’t just memorising but truly learning, you need to validate its performance. This is Step 5, validating the solution.
Machine learning relies on three key datasets to assess how well a model is learning:
- Training Set: These are the examples the model learns from, think textbooks and simulations.
- Validation Set: This separate set checks whether the model is generalising well, just like a mock example to catch overconfidence.
- Test Set: The final exam. This is data the model has never seen before, used to evaluate its true performance.
Using all your data for training is like giving a student the test answers ahead of the exam. They’ll get a straight grade A, but it won’t tell you if they’ve actually learned anything.
That’s why we hold back data for validation and testing: to ensure the model isn’t just memorising, but truly understanding. It’s not about passing the test, it’s about being ready for the task or mission.
Feature Engineering’s Catch: What is really relevant to the task or problem
Feature vectors powered traditional ML. They transformed messy, real-world data into structured inputs the model could learn patterns from. But we already hinted earlier that this process came with a catch: it relied heavily on human judgement to decide what was relevant and what wasn’t. And sometimes, that wasn’t always obvious.
Shall we do another Star Trek analogy? Yes! I hear you say. Alright then, let’s take the case of Dr. Crusher analysing a patient (data point) aboard the USS Enterprise:
| Attribute | Relevant to diagnoses? |
|---|---|
| Tricorder Scan - Pulse irregularity | YES |
| Tricorder Scan - Cortical Node Activity | YES |
| Starfleet uniform colour | NO (Unless it’s red and has many pips…) |
| Counsellor Deanna Troi’s Psychological Evaluation | NO (Rich in insight, however hard to quantify) |
Troi’s records might contain emotional states, behavioural observations, or empathic impressions. All valuable, yes, but difficult to encode into a neat numerical format. That’s the challenge: deciding what matters, even when the signal is subtle or subjective.
Each of these features is a clue. But in traditional ML, it’s up to the human to decide which clues matter. And sometimes, even Dr. Crusher’s instincts might miss a subtle signal buried deep in the data.
The Trouble with Manual Feature Engineering: It’s not always obvious
That’s the challenge of manual feature engineering: you need to know what’s relevant before the model can learn. But what if the real signal isn’t obvious? What if it’s hidden in the subspace noise?
Imagine Dr. Crusher again (or maybe even young Dr. Julian Bashir) trying to predict disease risk from medical records. They might write little rule snippets to extract features, such as:
“If blood pressure > X, flag as high risk.”
But what if the real signal is buried in the Dr.’s notes? Or in a subtle combination of test results?
This is where traditional ML methods hit their limits. They required human intuition to decide what mattered. And that intuition could be flawed.
End-to-End Learning: Let the model decide
Modern approaches flipped the textbook. Instead of manually crafting features, we let the model learn them from the raw data. This is the core of end-to-end learning, the model doesn’t just make predictions, it learns what features are relevant as part of the process.
It’s like giving a Cadet full access to the ship’s sensors and letting them figure out what readings matter most. No pre-filtering just raw input and intelligent inference.
The result? Models that are more accurate and more flexible, and often capable of uncovering patterns humans might miss. End-to-end learning shifts the burden of feature engineering from the human expert to the algorithm, allowing it to discover subtle signals hidden deep in the data nebula.
Hybrid Missions: Combining supervised and self-supervised learning
Not every mission is purely supervised or fully autonomous. Some of the most effective learning journeys blend both worlds, these are hybrid missions.
In semi-supervised learning, models begin by exploring vast unlabelled datasets using self-supervised techniques. They learn patterns, structures, and signals without explicit guidance. It’s like a Cadet analysing ship logs and sensor data to build intuition before receiving formal orders.
Once the model has built a foundational understanding, supervised learning steps in to refine its predictions using a smaller, labelled dataset. This hybrid approach is efficient, scalable, and increasingly common in real-world applications where labelled data is scarce but raw data is abundant.
By combining the autonomy of end-to-end learning with the precision of supervision, hybrid missions allow models to learn deeply, adapt quickly, and generalise effectively.
The Final Frontier: From regularities to foundation models
At its core, machine learning is about discovering regularities, patterns that hold across examples. These patterns can be subtle, surprising, and sometimes invisible to human eyes. But they’re the key to building models that truly understand.
Whether it’s spotting early signs of cancer in medical images or detecting fraud in financial transactions, ML models are increasingly able to see what we can’t. And that’s not just useful, it’s transformative.
We started with handcrafted features and simple models. We moved to end-to-end learning and deep architectures. Now, we deploy foundation models trained on galaxy-scale data, capable of adapting to new tasks with minimal effort (few-shot, zero-shot).
But no matter how advanced the ship or model, the mission always begins with a question. The tools may evolve, but the mission remains the same: to learn, to generalise, and to boldly predict what no model has predicted before. Make it so!
Curious how ML could power your next mission? Let’s talk.
Missed the other blogs?
Catch up on our Machine Learning series below:
Let's work together
We would love to speak with you.
Feel free to reach out using the below details.