Today, I’d like to tell you the story of my weekend, which involved a team-matching algorithm and some AI.
I developed and teach UW-Madison’s Computer Science Capstone course, which is a phenomenal course (I’m definitely biased) in which we pack in as much real-world experience for our students as one can in an academic setting. In my course, we partner with companies like Amazon / Shopbop, Google, and dozens of other companies who provide a project to work on for the semester and mentors for our students to learn from along the way. I take the first weekend of each semester to form teams of 4-6 students based on students’ skills and which companies they are interested in working with. It’s a basic optimization problem, one that can be solved a dozen or more ways with infinite outcomes, depending on what you’re optimizing for.
Capstone Teaming – Self-Selection
Though I teach computer science, I’ve always taken great care to form teams the way I formed them as a manager at places like Capital One and Opower: via self-selection. I started out letting students run through a pure self-selection process, which, in a nutshell, allows them to choose what they work on and who they work with. We used to be able to form about 15 teams in 2 hours across 80 students, most of which were happy with the teams they matched with.
But, as the class got bigger (I currently have 125 students enrolled) and the job market worse, self-selection began to break down. Students became selfish and would refuse to choose a team based on anything other than what they wanted. The process also took a lot longer, so we had to move to another strategy.
My Capstone Teaming “Algorithm”
So I started having students stack rank their project choices, tell me which friends they’d love to have on their teams, and link out to their resumes before our first class period. Then, I would spend the weekend creating teams. If fewer than 4-6 students ranked one partner as their #1 choice, they would all get that partner.
There were some popular partners who would have students clamoring for a spot, while others got very little interest. We still needed to form teams for each. So, in what can only be described as more art than science, I would pull people who wanted the oversubscribed partner onto teams that they wanted slightly less. Eventually we would get fully-formed teams and most people would be happy. I might have one person every semester become distraught and drop the course. This was pretty on par with what happened with self-selection too. When you have a large number of people, free will, and limited options, some people may end up disappointed. That’s a real-world lesson in and of itself.
AI Experiment #1: Gemini
So, this year I thought, “hmmm, I wonder if I could get AI to build me better teams faster?” I was almost certain the answer was, “YES!” So, I tried it and I started with my go-to platform right now: Gemini. I uploaded the spreadsheet holding all of our partners, the students’ names, their rankings, their resume links, their requests to work with people, and various other hopes and dreams. Here’s the first query I sent:
Using this spreadsheet, can you form teams of 4-6 students for each company/project listed across the top in line 2? Use the students in the first column from rows 7-120. Note that they have stack ranked their choices from 1-22 (one being the one they want most and 22 being the one they want least). Create the teams such that the largest number of students get their highest numbered choice possible. Please generate the output as a list of Company/Project, then student names including what they ranked that choice. One more thing, three students are colored purple in row A, those are UX students and only one can be on any given team.
And this, is what Gemini spit out.
This went on and on for “all of the companies on my list”: Alli Health, Banyan Water, Bedrock Analytics, Blockless, Bridgely, etc. And it looks really good…except for one thing.
NONE OF THESE ARE COMPANIES WE’RE WORKING WITH AND NONE OF THESE ARE STUDENTS IN MY CLASS.
This output is a total fabrication. Not to worry, we all know AI is capable of such things, so I said:
OK, just you’re making up the student names and the companies. I want you to use those that are in the document. The companies are listed in Row 2 and the Students are listed in column A.
To which it replied:
New companies, new students, none of them real.
AI Experiment #2: Claude
So, my expectations were extremely low at this point, but I thought I’d try another service. And I gave Claude the exact same first prompt and spreadsheet.
And Claude sat there and very carefully considered all the things.
It started out with:
And the outcome I got was nothing short of magical. I changed the student names below to protect their privacy.
And it goes on for 22 partners and 22 teams, each with a viable student mix.
I was able to then go through and have it update the teams to pay attention to which students wanted to work together and things got better. Later I asked it to optimize for giving the greatest number of students their first choice. And the results got even better.
Did I Use The Outcome?
Did I use the outcome of Claude’s algorithm to match teams this year? Yes and no. I actually did think Claude’s solution was better than the one I came up with, so I used it as the basis for our team matching. However, there were some enhancements that needed to be made.
For example, one of the partners wanted a UX student, so I did some switching to get that student not only her first choice, but also to give the partner a skillset they needed. The passionate student who wanted to work on an environmental project? I moved them onto our one environmental project. The two hispanic guys who wanted to work together and who did not get their overly-popular first choice, whilst a bunch of white guys did? I made some changes there. AI is just using the data we give them and, it seems, it still has its biases. In fact, I made quite a few changes taking into account students’ interests and background.
But, I am really happy with the outcome. More students this semester got higher ranked choices than ever before and I team-matched faster than ever before. For that, I owe Claude some thanks. I have no idea why Gemini failed so hard, but that just goes to show you, machines make (a lot of) mistakes too.