So now that our first alpha test in way too many years is over--at least, for now--I think a post mortem review of the process would be helpful for anyone undertaking a similar project for their own Age/game level.
First off, a huge thank you to everyone who joined in the test and provided feedback. I was often impressed by the depth of your analyses, even if, yes, you couldn't solve the puzzle. Fyi, only one person managed to solve the puzzle, and this was through luck. (Okay, so there were those of you who hacked the game files to get the answer. ) At any rate, we'll be fixing things to guide players towards the right answer in the next version.
Also--if you applied as a tester but got stuck at some point in the process--i.e. you sent us an e-mail, but we never responded--then please send another e-mail to email@example.com and we'll guide you through the rest of the steps. We made a number of mistakes along the way, and though we tried to make sure that we got everyone, I don't feel 100% sure.
Finally, if you played the game and haven't yet responded to the questionnaire (you can find the link in the same e-mail as the game download links), what are you waiting for? The game isn't going to improve itself. ;)
Okay, on to the post mortem review.
When we started off the alpha testing process, we simply had a bunch of loose testing documents set up on Google docs with spreadsheets and letters. Basically we just planned to e-mail links to whoever signed up. But the number of people who joined the test (several dozens) combined with me moving house and losing internet connection for awhile made the manual approach fall flat on its face. I tried sending mass e-mails, but it was difficult because everyone was at a different stage in the sign up process and I wasn't sure who had been taken care of yet; the done-by-hand lists took too much effort to maintain. Paulo finally solved the problem by creating a bot that automatically flagged e-mails with the appropriate "Sign me up for alpha testing" title, then created links between the documents which shunted people through the sign up process automagically. Muuuch better. He can explain this process better than I can, so I'll let him go over it in another post. :)
Another pitfall included the language barrier (I don't speak French, though fortunately several other Ilathidians do), which prevented several unfortunate people from getting prompt replies to their requests for help. If anyone is interested, we are looking for a French translator; excellent spelling and grammar are a must.
Then too, the download site we used for the game was not particularly "clean," leading at least one person to believe that they had to pay to get the game. I recommend choosing a very simple, user friendly download site without any unnecessary content on the download page.
Finally, this may seem obvious, but note in the alpha testing announcement what platform the game is currently running on. Mac, Linux, Windows--it's kind of important, y'know? ;)
Okay, that's it for now. On to the next bunch of fixes!