Postdoc simulation analysis (spoiler alert: job insecurity is bad)

Once again I’ve been working with the academic job security simulation again.  Yesterday I’d finished altering the research funding model so that our poor agents no longer lived in a world of government largesse where population increases are always matched by an increase in funding to keep grant acceptance rates at 30%.

After tweaking things a lot last night and earlier today, I found that a funding level set at an initial 30% with a 2% increase per timestep led to research output levels very close to the previous version of the model.  The proportion of grants funded slowly drops over the course of 100 timesteps, heading from that starting 30% down to about 17% at the end of an average run.

I also added a simple retirement mechanism to this version: after 40 semesters, agents start to think about retirement and have a fixed chance (20% at the moment) of leaving the sector forever.  The result of this is a significant rise in the return-on-investment measure as the senior academics start to leave the sector; seems we had a lot of senior academics coasting along without producing much in the way of research!  Compared to the previous version, the older academics produce significantly less research-wise — I’m presuming this is because the rich-get-richer aspect of the increasingly competitive funding environment leads to a larger proportion of failed applicants deciding to bow out of the rat-race altogether.

Having taken a brief look at all this I decided to test the feedback given to me at Alife XV.  In the initial simulation, promotions had a huge positive impact on research output regardless of whether they were made entirely at random or based on research quality.  Several people at the conference suggested that this may no longer be the case if I implemented a more constrained funding system.

So, I ran the simulation 800 times across a range of parameter values with limited funding and retirement mechanisms both turned on.  I then used my old pal, the software called Gaussian Emulation Machine for Sensitivity Analysis (GEM-SA), to crunch the numbers and come up with a statistical model of the agent-based model, and then ran that 41,000 times.  The final output of interest is the total research produced across the agent population at the end of the simulation.  The analysis looks like this:

gemsa1-cropped

Turns out my colleagues were onto something, which I expected (and hoped for, because otherwise that might mean the simulation might have some problems).  In this version of the sim altering the chances of promotion for postdocs does little on its own, accounting for only 0.11% of the output variance.  This factor interacts with the level of stress induced by impending redundancy, however, and this interaction accounts for 11.03% of the output variance.

The largest effect here is driven by Mentoring levels — the amount of research boost given to newly-promoted postdocs.  Second-largest is the stress caused by looming redundancies.  This is a significantly different result from the previous version of the simulation — I’ll run a parameter sweep of promotion levels later as well, to get the complete picture.

For the sake of completeness, here’s the graph of the main effects produced by GEM-SA:

gemsa2-cropped2

Tomorrow I’m hoping to do a similar analysis, but this time leaving Mentoring at a lower, constant level and varying a slightly different set of parameters.  My poor laptop needs a break for a little while, it’s pumping out crazy amounts of heat after all this number-crunching.

My other, larger task is to come up with a way to measure the overall human cost of this funding/career structure.  I think I can make a good case at this point that job insecurity is not great for research output in the simulation, given that across many thousands of runs I’ve yet to find a single one in which insecure employment produces more research for the money than permanent academic jobs.  I’d to be able to compare scenarios in terms of human cost as well, so perhaps taking a look at total redundancies after 100 semesters as the final output for some analyses might give me some ideas.

That aside I think I’ve made a decent start on an extension of the conference paper.  Thanks to all those who came to the talk in Mexico and gave me some useful feedback!

 

Tagged , , , ,

Returning to the postdoc simulation

I’ve been doing some tests on my postdoc simulation today. As suggested by colleagues at Alife XV I’ve implemented a funding system in which the total available funding increases at a lower rate than the population, leading to increased competition.

Total research output under these conditions does increase pretty significantly — however, return-on-investment remains negative, meaning we still would get more for our money by hiring half as many permanent researchers instead of postdocs. Postdocs are still the group producing the lion’s share of the actual scientific work, while permanent academics nearly all of their research time to grant-writing.
 
The return-on-investment is less negative than under the previous funding condition, but bear in mind the simulation currently doesn’t account for redundancy payments or training costs for new postdocs.  In these runs results were showing a return of -2.5 papers per unit of funding invested as compared to a postdoc-free scenario; in the unlimited-funding condition with the same settings, the figure averaged -3.6/unit.  Redundancies were higher in this condition, about 150 more each run than in the unlimited funding condition.  This could change significantly, however, depending on the final formula I use for year-on-year research budget increases, given that postdocs’ fates are directly tied to how much research money is available.
 
The big question is whether under this condition we still see no improvement in research output or return-on-investment when candidates for promotion to permanency are selected by quality rather than randomly. At this early stage there’s little difference — I’ve only done a few runs, but non-random promotions have not demonstrated a significant difference from random ones in either total output or ROI.  We’ll see if that changes when I do a larger sequence of runs.

My next test after that will be to try this version of the simulation with much longer runs to see if things stabilise at all, or whether the uncertainty introduced by the high-turnover postdoc population continues to drown out any attempts at rewarding high achievers with more grants.  We’re already looking at 50-year runs here, though, so if after two centures of terrible job security we see hugely better cost efficiencies I’m still not sure that’s a massive win for the postdoc side of things.  But I suppose that rests on whether you care about the human costs or not.

There’s a lot of tweaking to be done so these are very early days, but it’s an interesting first result.
Tagged , , , ,

Diversions: Game Dev Update

-WARNING: MORE GAME NERD STUFF-

So in my evening hours I’ve been working away on the roguelike game development project.  For the most part I’ve been really pleased with how well it’s trucking along — I’ve seen a number of other roguelike projects out there using Libtcod that fizzled out way before adding this amount of content, so I feel confident I can finish this project.

Having said that, it’s requiring a ton of changes in my coding practices.  Even in just a couple of weeks of development time, my game has expanded to the point where every addition has a large number of potential interactions with all the other game components, so I’ve had to give in and embrace version control.  Every change is archived on my own machine, then on Google Drive, and finally uploaded, checked, and merged into the master code branch on GitHub.  I’ve never bothered to do this before, but with this project (where just adding a new potion can cause weird stuff like making monsters stop moving completely until I find some minor malfunction) I’ve found I need to make it as simple as possible to back up a step when something goes strange.

I’ve added a few major gameplay additions since my last post.  One of them appears right at the start of the game — you can now choose a character role when you start the game.  Fighters are focused on close-range combat, Knights carry a shield which gives them great survivability, Archers are great at long-distance but weak up close, and finally Wizards are incredibly fragile but start with powerful spells.  Here’s a wizard who’s just managed to turn a wolf to stone:

update5

You might notice that I’ve been experimenting with the ASCII visual presentation too — the plan is to have some colour schemes for each set of dungeon levels, and each set will have particular characteristics and terrain hazards too.  I’ve also changed the lighting slightly, heightening the contrast between explored and unexplored squares, and I’ve changed the font to a 16×16 version of Terminal that I think is a bit easier on the eyes.

Since I’ve introduced Wizards, I’ve also introduced the beginnings of a magic system — although there’s a lot left to do in this area.  I’ve followed the lead of other roguelikes and developed a system of wand items — these drop randomly in the dungeon or from certain enemies, and give you a limited number of uses of a specific spell.  I’ve added eight wands, each with two varieties, as a first test of the magic system and how it affects play.  Later I’ll be adding a system for Wizards to learn spells permanently, a resource system to restrict usage, and more detailed stats for players and monsters to make certain spells more effective against particular opponents.

I particularly enjoyed making the Petrification effect — it turns the monster into stone, of course, which then blocks visibility and movement.  So you can use it tactically to block off other monsters and restrict their movement or vision.  When you tire of it you can destroy the statue and it crumbles into rocks (which you’ll be able to use as ammo for a sling, once I make that).

As for monsters, I’ve added a few of those, but more importantly I’ve added a random mutation system for monster creation.  When a monster is generated, there’s a small chance that the monster will be mutated either one or two times, and each mutation will add statistical bonuses and special abilities, as well as alter the monster’s name and display colour.  I’ve only added six mutators so far, but that’s already enough to significantly increase the game variety.  Honestly I probably spend more time play-testing than I really should when I have coding time — but I think that’s a good sign if I’m already finding it fun to play!  The mutators will be even better once I add more monster types — these are taking time as each monster type has it’s own AI function which takes a lot of testing to get right.

Here’s a Fighter who’s run into a Hellfire Troll, a significantly stronger mutation of the already difficult Troll:

update1

All told it’s been a productive week.  My goal for the weekend is to jazz up the dungeon environment significantly, first by setting up the visual themes and names for each level subset, then developing a system for adding terrain features, some of which will be interactive (think stuff like lakes, lava flows, grass, fungus, rock pillars, etc.).

After that there’s some major tasks in my development plan:

  • Flesh out the magic system (which will need a bunch of interrelated items and systems to go with it)
  • AI and game systems for allied characters (in some circumstances players will be able to raise some troops, by resurrecting dead monsters or brainwashing living ones)
  • Methods for random generation of items

Again none of these are new ideas — my design goal here is simply to make a complete roguelike in the classic style, with just a bit more playability and transparency.  Once it’s done I’ve got some ideas for game #2 — but I’m not allowing myself to think much about that now!

Speaking of roguelikes in the classic style, thanks to some dedicated roguelike fans out there you can actually still play the original Rogue on modern computers.  It’s still quite playable, albeit mercilessly hard — winning Rogue is a real accomplishment.

You can also still play Moria — the seminal early roguelike based on Tolkien which later spawned Angband.  Really though I’d recommend trying Angband if you want to try a classic roguelike — it’s much more user-friendly, has lots more content, and has a graphical mode.

Tagged ,

A Diversion: A First Attempt at Game Development

–WARNING:  LOTS OF WORDS INCOMING–

As my friends and colleagues are aware, my favourite hobby outside the academic sphere is gaming.  In particular I have a fondness for very difficult games that require some tactical thinking to survive — so I’m a big fan of roguelikes.

For those of you who don’t know what a roguelike is, I’ll share with you a too-lengthy post I made on Facebook by way of explanation:

My game is a ‘roguelike’, a genre named for a very popular game in this style from 1980 called Rogue. There’s some debate about how to define roguelikes precisely, but generally they’re characterised by:


1) Turn-based gameplay — you take a turn, then the enemies. You can take as long as you like to think about each action and its consequences.


2) Randomly-generated levels — often you dive into a very long, multi-level dungeon full of monsters, items and fiendish traps, and every level is randomly generated each time you play, meaning you never run out of levels to play and no play-through is ever the same. In my game dungeons are generated using BSP trees which is a pretty common method.


3) Permanent death — if your character dies, that’s it, you have to start over from the beginning! What’s more, most games (including mine) automatically save the entire game state after every move, and overwrite your saved game when you reload a save, meaning that you can’t ever go back to a previous turn. Every action has permanent consequences.


4) Text-based graphics are usually a feature of every roguelike in the classic style, even today in 2016. Graphical modes are normally an option nowadays too, but many people prefer text mode (including me) because once you get used to it the text actually makes it much easier to determine exactly what’s in front of you at any given time, which is important because…


5) …while the graphics are often simple, the gameplay is normally quite complicated. Hundreds of items, monsters, and environmental features are in these games, all of which can interact in tons of different ways — not needing to make fancy animations for everything means developers can go nuts with the gameplay systems.

Anyway those are the main features you see in the genre — most of the major games in the genre are completely free, so I recommend trying them. The site RogueBasin has links to the five ‘major roguelikes’ right on the front page — ADOM, Angband, Crawl, NetHack and ToME. In my opinion Crawl or ToME are probably the easiest places to start, given they have pretty nice visuals and decent tutorials for new players. Angband is a long-time classic and has a decent graphical mode included as well. NetHack is famous for having insanely detailed game systems and item interactions, but is a bit overwhelming for newcomers.

My game is very early in development, although the core gameplay systems are all functional (random dungeons, character progression, combat, item and magic systems) and it’s fully playable. Not bad IMO given I only started making it five days ago, but there’s a long way to go yet. My goal is to add at least one new gameplay feature, monster, item, etc. every day for the next couple months so I can gradually build up a complex but balanced gameplay experience.

Given that this is my first-ever foray into making a complete game — I don’t count my failed attempts to write code for the GameBoy Advance back in 2005 — I’m pretty pleased with my progress, and it’s certainly been an enjoyable and challenging way to spend a few days of my summer holiday.

I started off by following the Complete Roguelike Tutorial using Python 2.7 with Libtcod — this gets you a complete, playable prototype (albeit very simple).  Since then I’ve added quite a few major features, each of which has required learning a bunch of new algorithms or techniques:

  • A* pathfinding for monsters
  • Random dungeon generation via BSP trees
  • A graphical version (using free 16×16 tiles made for Angband)
  • A more complex combat system including dicerolls and critical hits
  • Damage-over-time (poison attacks)
  • A complete ranged combat system including ammo tracking and relevant attributes for the player and enemies
  • Several new AI routines designed for specific enemies, i.e.:
    • Wolves that call for their pack mates when in trouble
    • Snakes that stay at a distance and spit poison
    • Enemy archers who try to get the drop on you

Now that the core gameplay systems are mostly there, and the return to academic life is looming, I’ll be slowing down significantly from here.  My goal is to build on this foundation bit by bit over the coming weeks and months, until I have a cohesive 20-floor dungeon-crawl experience with quite a few dozen monsters, items and weapons to discover.  This is a bit of an experiment for me, and I may not release this game to the larger world and instead bank this experience for a better follow-up project.  Having said that, once it’s in a more complete state I’d welcome any interested play-testers!

I should note that if I do end up releasing this it’ll be completely free, and probably open-source, assuming my code isn’t too embarrassing.

Since this is ostensibly an academic blog I should say that part of what pushed me to finally take the plunge and try to make something like this was actually some of the students I advised last academic year.  A large proportion of my advisees were game development students, and I wished I’d had more domain-specific knowledge to help them through certain problems.  Now at least I can tell them where to look when I hear from students who are stuck on pathfinding, AI issues, etc.!

Before I leave you, a screenshot of the fabulous ASCII graphics I’ve been staring at for hours on end:

poison_screenshot9-cropped

And for the ASCII-shy, a screenshot of the graphical version:

tiles_screenshot6_cropped

More to come later — the graphical version is very much a side-project at the moment, so expect that to be spruced up as time goes on.  I’m planning to switch to the stylishly lo-fi tiles of Oryx Design Lab — they’re simple, visually clear, and as they’re uncoloured I can tint them a million ways to represent the requisite dozens of variations of every imaginable monster that roguelikes generally have.

If you think I’m exaggerating — NetHack variant Slash ‘Em Extended boasts 12,221 monster species.  I think I’ll stop long before the 12,000 mark though.

That’s enough out of me for today — I’ll post updates on here every so often, if I find any particularly interesting techniques or come up with a version polished enough to distribute.

 

 

Tagged , ,

Alife XV Presentation

I’ve been attending Alife XV all week in extremely hot and sweaty Cancun, Mexico.  Yesterday I gave a talk on my paper with Nic Geard and Ian Wood titled Job Insecurity in Academic Research Employment: An Agent-Based Model.

I really enjoyed giving the talk — I spent a great deal of time beforehand thinking about how to introduce the work in proper context, and in the end I felt it worked reasonably well.  I had some great questions which raised important points that we’ll be taking into account in the next iteration of the model.  I’ve had a number of colleagues share their enthusiasm about the topic since the talk, so I’m really pleased and hopeful this work will keep advancing.

Thinking about the feedback I received, I think the most important next step is to develop the competitive funding aspects of the model in more detail:

  • Instead of an optimistic world with research funding that scales with population, have a pot of funding which grows at a slower rate, leading to a gradually more selective competitive process
  • Test possible implementations of more varied grants — larger/smaller grants which can produce more postdocs, grants of a longer duration, etc.
  • Possibly too ambitious for the near future, but implementing a system of teaching quality/student funding which also requires time allocation from the agents would be an interesting direction to take this

I’ve uploaded the presentation slides, and the final published paper is available here.  The full Alife XV Proceedings volume is available open-access via MIT Press.

Tagged , , ,

So, Vice-Chancellor, now what?

Paraffinalia

So the UK is leaving the European Union and it is time for the Vice-Chancellors to panic. Before `university leaders’ tell university staff, students, and the public who pay their inflated salaries that “we’re all in this together” or some variation thereon, begging us to help get them out of this mess, they might like to think on their own conduct.

View original post 292 more words

Funded PhD opportunity at Teesside

Fully-funded PhD opportunity available! I’m looking for someone interested in working on agent-based modelling for healthcare applications. No fees and £20K stipend. These are four-year positions and you will be asked to contribute up to six hours per week of teaching (tutorials/demonstration only, no lectures), which is more work but also good for the CV. Click here and filter under ‘Computer Science’ to see my project.  For more about me, check out the various pages on this blog or my staff profile at Teesside.

Project description: This research will focus on the application of Agent-Based Modelling techniques to human social systems, with particular emphasis on digital health applications. In the context of public health, agent-based models can help us understand the complexities of health policy implementation and service delivery by modelling the multiple interacting processes underlying the health system. These models will investigate challenges in health and social care service delivery across a variety of spatial and temporal scales – from short-term studies of demands on accident and emergency services, to longer-term explorations of the pressures facing social care over the next several decades. Our multi-disciplinary team will work with members of the School of Health and Social Care here at Teesside, along with external collaborators and stakeholders. The project would be suitable for a graduate with a background in Computer Science, Artificial Intelligence, Statistics or Complexity Science with an interest in Public Health/Healthcare applications.

ACADEMIC FRIENDS: Please tweet/share this as widely as you can!

Tagged , , , ,

The ‘invisible army’ of elderly carers

Some of you may have seen my paper with Jason Noble, Jason Hilton and Jakub Bijak from 2013 called Simulating the cost of social care in an ageing population.  The paper presents an agent-based model of informal social care in the United Kingdom.  Our virtual agents live in a simulated UK, and try to live out their lives — moving around, working, starting families, etc. — and when their family members need help due to illness, they try to contribute their time to help out.

Model results showed that, surprisingly, retirement age has a strong impact on social care costs across the population.  When the retirement age was raised, there was a net increase in tax revenues up to a certain point, but beyond that critical limit social care costs began to rise.  The model seemed to indicate that an unexpectedly large number of elderly people were providing informal care to their spouses or other loved ones, and so putting them back into the workforce actually led to increased demand for state-funded formal care for those left at home, increasing the cost to society overall.

I’ve just noticed a news posting from Age UK from last month which is pretty relevant to this:

New figures released this morning by Age UK show there is an army of carers amongst the oldest in our society, who are between them saving the health and care system a massive £5.9bn a year by providing unpaid care.

Over the past 7 years the number of carers aged 80 and over has rocketed from 301,000 to 417,000, an increase of nearly 39%. Now 1 in 7 people aged 80 and over provide some form of care to family or friends.

Furthermore, over half (144,000) of carers in this age group who are caring for someone in their home are doing so for more than 35 hours a week, while a further 156,000 are caring for more than 20 hours a week. As our population continues to age it is estimated that there will be more than 760,000 carers aged 80 and beyond by 2030.

I’m the first to admit that I’m a bit of an outsider when it comes to gerontology and the study of social care in detail, so it’s possible that this study isn’t telling us much that’s new.  It’s news to me, however, and I’m glad to see that our model showed us some interesting results that turned out to be reflective of reality, despite the necessarily simplified nature of the model’s systems.

Now that there’s some solid data out there about this ‘invisible army’ of older carers, I think it may be time to revisit this model and investigate this aspect more fully.  Caring for someone 35 hours a week or more is exhausting work for anyone, let alone someone over 80 years old who should be enjoying a dignified retirement.  Perhaps we can use agent-based models to investigate policies that could take some of this burden away from our older population.

Paper accepted to Alife XV

I’m pleased to say that the paper I’ve been going on about now for some time, titled Job Insecurity in Academic Research Employment: An Agent-Based Model, has been accepted to Alife XV in Cancun this summer.  I’m currently working on some revisions to the paper to account for some helpful suggestions from the reviewers — as soon as the final camera-ready preprint is available I’ll post it here and the usual places (ResearchGate, Academia.edu, etc.).

Hope to see some of you in sunny Mexico come July🙂

Tagged , , , ,

Paper Submitted To Alife XV

I’m happy to report that I’ve recently submitted a first paper on the postdoc simulation I’ve been plugging on these pages for some time.  I’ve been working in collaboration with Nic Geard of the University of Melbourne and Ian Wood, my officemate at Teesside.

The submitted paper is titled Job Insecurity in Academic Research Employment: An Agent-Based Model.  Here’s the abstract:

This paper presents an agent-based model of fixed-term academic employment in a competitive research funding environment.  The goal of the model is to investigate the effects of job insecurity on research productivity.  Agents may be either established academics who may apply for grants, or postdoctoral researchers who are unable to apply for grants and experience hardship when reaching the end of their fixed-term contracts.  Results show that in general adding fixed-term postdocs to the system produces less total research output than adding half as many permanent academics.  An in-depth sensitivity analysis is performed across postdoc scenarios, and indicates that promoting more postdocs into permanent positions produces significant increases in research output.

The paper outlines our methodology for the model and analyses a number of different sets of scenarios.  Alongside the comparison to permanent academic hires mentioned above, we also look closely at unique aspects of the postdoc life cycle, such as the difficult transition into permanent employment and the stress induced by an impending redundancy.  For the sensitivity analysis we used a Gaussian process emulator, which allows us to gain some insight into the effects of some key model parameters.

The paper will be under review for the Alife XV conference very shortly, so I don’t want to pre-empt the conference by posting the full text here.  If — fingers crossed — it gets accepted, I’ll post a PDF as soon as it’s appropriate.  If you want a preview or are interested in collaborating on future versions of the model, please get in touch!

Tagged , , , , , ,